AnyTalk：基于视频生成模型的任意角色语音动画

摘要

我们提出了 AnyTalk，一种无需任何动画数据即可为任意角色生成 3D 语音动画的新方法。现有基于音频驱动的 3D 语音动画方法依赖于特定角色的训练数据或繁琐的绑定/重网格化，而 AnyTalk 通过利用在大规模视频数据集上训练的最新视频扩散模型规避了这些限制。我们首先通过角色特定微调（CsF）技术将预训练的视频扩散模型适配到目标角色。通过将 3D 角色的渲染图像与置零的音频嵌入（表示“无运动”）配对进行微调，我们在保留大规模视频扩散模型运动先验的同时，消除了对动画数据的需求。随后，我们通过所提出的优化过程估计融合变形（blendshape）参数，将生成的说话头视频转化为 3D 语音动画。AnyTalk 能够在多种人脸网格和融合变形配置下实现口型同步动画，显著减少人工操作和数据需求。我们进一步将 AnyTalk 蒸馏为精简网络 AnyTalk_{RT}，从而提升可用性并实现实时性能。通过利用说话头视频生成技术，我们的方法拓宽了音频驱动语音动画技术在任意角色上的应用渠道。代码公开于 https://serin-yoon.github.io/projects/anytalk/。

English

We present AnyTalk, a novel method for generating 3D speech animations for arbitrary characters without requiring any animation data. While existing audio-driven 3D speech animation methods rely on character-specific training data or laborious rigging/re-meshing, AnyTalk circumvents these limitations by leveraging recent video diffusion models trained on extensive video datasets. We first adapt a pre-trained video diffusion model to a target character through our Character-specific Fine-tuning (CsF) technique. By fine-tuning on rendered images of the 3D character paired with zeroed-out audio embeddings (representing "no motion"), we eliminate the need for animation data while preserving the motion prior of large-scale video diffusion model. We then uplift the resulting talking-head video into a 3D speech animation by estimating blendshape parameters through a proposed optimization process. AnyTalk enables lip-synced animations across diverse face meshes and blendshape configurations, significantly reducing manual effort and data requirements. We further enhance usability by distilling AnyTalk into a streamlined network, AnyTalk_{RT}, thereby enabling real-time performance. By leveraging talking-head video generation, our method broadens access to audio-driven speech animation technology for arbitrary characters. The code is publicly available at https://serin-yoon.github.io/projects/anytalk/.