麒麟：從野外影片生成動物動作

摘要

理解動物動作是建立動物行為與生物力學模型的基礎，然而受限於高品質動作資料的匱乏，此領域的進展遠落後於人類動作研究。人類動作可在受控環境中捕捉，但對大多數動物物種而言並不可行，因此現有資料集規模小且僅限於特定應用領域，限制了動畫等下游應用的發展。為了解決這項挑戰，我們提出了 Kirin，這是一個從影片重建動作、大規模學習動作先驗，並生成能直接應用於動畫資產的逼真動作之框架。我們利用大量野外動物影片，重建 3D 動作序列並與文字描述配對，建立了 AiM3D——首個為四足動物提供對齊的影片-文字-動作三元組的大規模資料集。在此資料集的基礎上，我們開發了一個視覺引導的動作生成模型，該模型以文字與圖像為條件，引導生成跨多種動物物種的逼真動作。最後，我們利用現成的影像轉 3D 模型，將生成的動作自動套用至 3D 網格並完成綁定與動畫製作，產出可直接渲染的動物動畫。我們的資料集與框架共同為大規模、以文字與影像為條件的動物動作生成及動畫建立新的基礎。專案頁面：https://kirin-ani.github.io/。

English

Understanding animal motion is fundamental to modeling animal behavior and biomechanics, yet progress in this area lags far behind human motion research due to the scarcity of high-quality motion data. While human motion can be captured in controlled environments, it is impractical for most animal species, resulting in small, domain-limited datasets that restrict downstream applications such as animation. To address this challenge, we introduce Kirin, a framework that reconstructs motion from video, learns motion priors at scale, and generates realistic motion that can be directly applied to animated assets. Using large collections of in-the-wild animal videos, we reconstruct 3D motion sequences and pair them with captions to create AiM3D, the first large-scale dataset offering aligned video-text-motion tuples for quadruped animals. Building on this dataset, we develop a visual-guided motion generation model that conditions on both text and image to guide the generation of realistic motion across diverse animal species. Finally, by leveraging an off-the-shelf image-to-3D model, we automatically rig and animate 3D meshes using generated motion, producing ready-to-render animated animals. Together, our dataset and framework establish a new foundation for large-scale, text and image conditioned animal motion generation and animation. Project page: https://kirin-ani.github.io/.