麒麟：从野外视频生成动物动作

摘要

理解动物运动是动物行为与生物力学建模的基础，然而由于高质量运动数据的匮乏，该领域的进展远落后于人类运动研究。人类运动可以在受控环境中捕捉，但对于大多数动物物种而言这并不现实，由此产生的数据集规模小且局限于特定领域，制约了动画等下游应用。为应对这一挑战，我们提出了Kirin——一个从视频中重建运动、大规模学习运动先验，并生成可直接应用于动画资产的逼真运动的框架。利用大规模自然场景中的动物视频，我们重建了3D运动序列并配以文本描述，构建了AiM3D——首个为四足动物提供视频-文本-运动三元组对齐的大规模数据集。基于该数据集，我们开发了一种视觉引导的运动生成模型，以文本和图像为条件，引导生成跨多种动物物种的逼真运动。最后，通过利用现成的图像转3D模型，我们使用生成的运动自动为3D网格进行骨骼绑定与动画驱动，产出可直接渲染的动物动画。综上，我们的数据集与框架为大规模、文本与图像联合引导的动物运动生成与动画制作奠定了新基础。项目主页：https://kirin-ani.github.io/。

English

Understanding animal motion is fundamental to modeling animal behavior and biomechanics, yet progress in this area lags far behind human motion research due to the scarcity of high-quality motion data. While human motion can be captured in controlled environments, it is impractical for most animal species, resulting in small, domain-limited datasets that restrict downstream applications such as animation. To address this challenge, we introduce Kirin, a framework that reconstructs motion from video, learns motion priors at scale, and generates realistic motion that can be directly applied to animated assets. Using large collections of in-the-wild animal videos, we reconstruct 3D motion sequences and pair them with captions to create AiM3D, the first large-scale dataset offering aligned video-text-motion tuples for quadruped animals. Building on this dataset, we develop a visual-guided motion generation model that conditions on both text and image to guide the generation of realistic motion across diverse animal species. Finally, by leveraging an off-the-shelf image-to-3D model, we automatically rig and animate 3D meshes using generated motion, producing ready-to-render animated animals. Together, our dataset and framework establish a new foundation for large-scale, text and image conditioned animal motion generation and animation. Project page: https://kirin-ani.github.io/.