動物如何舞動（在你未曾察覺之時）

How Animals Dance (When You're Not Looking)

May 29, 2025

作者: Xiaojuan Wang, Aleksander Holynski, Brian Curless, Ira Kemelmacher, Steve Seitz

cs.AI

摘要

我们提出了一种基于关键帧的框架，用于生成音乐同步且考虑编舞的动物舞蹈视频。从代表不同动物姿态的少量关键帧出发——这些关键帧通过文本到图像的提示或GPT-4o生成——我们将舞蹈合成表述为一个图优化问题：寻找满足特定编拍模式的最优关键帧结构，该模式可从参考舞蹈视频中自动估计得出。此外，我们引入了一种镜像姿态图像生成方法，这对于捕捉舞蹈中的对称性至关重要。中间帧则通过视频扩散模型进行合成。仅需六个输入关键帧，我们的方法便能生成涵盖多种动物和音乐曲目、长达30秒的舞蹈视频。

English

We present a keyframe-based framework for generating music-synchronized, choreography aware animal dance videos. Starting from a few keyframes representing distinct animal poses -- generated via text-to-image prompting or GPT-4o -- we formulate dance synthesis as a graph optimization problem: find the optimal keyframe structure that satisfies a specified choreography pattern of beats, which can be automatically estimated from a reference dance video. We also introduce an approach for mirrored pose image generation, essential for capturing symmetry in dance. In-between frames are synthesized using an video diffusion model. With as few as six input keyframes, our method can produce up to 30 second dance videos across a wide range of animals and music tracks.