Animate-X：具有增强运动表示的通用角色图像动画

Animate-X: Universal Character Image Animation with Enhanced Motion Representation

October 14, 2024

作者: Shuai Tan, Biao Gong, Xiang Wang, Shiwei Zhang, Dandan Zheng, Ruobing Zheng, Kecheng Zheng, Jingdong Chen, Ming Yang

cs.AI

摘要

角色图像动画，从参考图像和目标姿势序列生成高质量视频，在近年来取得了显著进展。然而，大多数现有方法仅适用于人物形象，通常在游戏和娱乐等行业常用的类人角色上泛化能力不佳。我们的深入分析表明，这种限制归因于它们对运动建模不足，无法理解驱动视频的运动模式，因此会将姿势序列严格施加到目标角色上。因此，本文提出了一种基于LDM的通用动画框架Aniamte-X，适用于各种角色类型（统称为X），包括类人角色。为增强运动表征，我们引入了姿势指示器，通过隐式和显式方式从驱动视频中捕获全面的运动模式。前者利用驱动视频的CLIP视觉特征提取其运动要点，如整体运动模式和运动之间的时间关系，后者通过提前模拟可能在推理过程中出现的输入，加强了LDM的泛化能力。此外，我们引入了一个新的动画类人基准（A^2Bench）来评估Animate-X在通用和广泛适用的动画图像上的性能。大量实验证明了Animate-X相对于最先进方法的优越性和有效性。

English

Character image animation, which generates high-quality videos from a reference image and target pose sequence, has seen significant progress in recent years. However, most existing methods only apply to human figures, which usually do not generalize well on anthropomorphic characters commonly used in industries like gaming and entertainment. Our in-depth analysis suggests to attribute this limitation to their insufficient modeling of motion, which is unable to comprehend the movement pattern of the driving video, thus imposing a pose sequence rigidly onto the target character. To this end, this paper proposes Animate-X, a universal animation framework based on LDM for various character types (collectively named X), including anthropomorphic characters. To enhance motion representation, we introduce the Pose Indicator, which captures comprehensive motion pattern from the driving video through both implicit and explicit manner. The former leverages CLIP visual features of a driving video to extract its gist of motion, like the overall movement pattern and temporal relations among motions, while the latter strengthens the generalization of LDM by simulating possible inputs in advance that may arise during inference. Moreover, we introduce a new Animated Anthropomorphic Benchmark (A^2Bench) to evaluate the performance of Animate-X on universal and widely applicable animation images. Extensive experiments demonstrate the superiority and effectiveness of Animate-X compared to state-of-the-art methods.