CharacterShot：可控且一致的4D角色動畫

摘要

本文提出CharacterShot，一個可控且一致的4D角色動畫框架，使任何設計師都能從單一參考角色圖像和2D姿態序列中創建動態3D角色（即4D角色動畫）。我們首先基於前沿的DiT圖像到視頻模型預訓練一個強大的2D角色動畫模型，該模型允許任何2D姿態序列作為可控信號。接著，通過引入雙注意力模塊和相機先驗，我們將動畫模型從2D提升至3D，生成具有時空一致性和視角一致性的多視角視頻。最後，我們對這些多視角視頻應用新穎的鄰域約束4D高斯濺射優化，從而獲得連續且穩定的4D角色表示。此外，為提升角色中心性能，我們構建了大規模數據集Character4D，包含13,115個具有多樣外觀和動作的獨特角色，並從多個視角進行渲染。在我們新構建的基準測試CharacterBench上的大量實驗表明，我們的方法優於當前最先進的方法。代碼、模型和數據集將在https://github.com/Jeoyal/CharacterShot公開提供。

English

In this paper, we propose CharacterShot, a controllable and consistent 4D character animation framework that enables any individual designer to create dynamic 3D characters (i.e., 4D character animation) from a single reference character image and a 2D pose sequence. We begin by pretraining a powerful 2D character animation model based on a cutting-edge DiT-based image-to-video model, which allows for any 2D pose sequnce as controllable signal. We then lift the animation model from 2D to 3D through introducing dual-attention module together with camera prior to generate multi-view videos with spatial-temporal and spatial-view consistency. Finally, we employ a novel neighbor-constrained 4D gaussian splatting optimization on these multi-view videos, resulting in continuous and stable 4D character representations. Moreover, to improve character-centric performance, we construct a large-scale dataset Character4D, containing 13,115 unique characters with diverse appearances and motions, rendered from multiple viewpoints. Extensive experiments on our newly constructed benchmark, CharacterBench, demonstrate that our approach outperforms current state-of-the-art methods. Code, models, and datasets will be publicly available at https://github.com/Jeoyal/CharacterShot.

CharacterShot：可控且一致的4D角色動畫

CharacterShot: Controllable and Consistent 4D Character Animation

摘要

Support