AnyTalk: 利用視頻生成模型的任意角色語音動畫

摘要

我們提出AnyTalk，一種新穎方法，用於為任意角色生成3D語音動畫，且無需任何動畫資料。現有的音訊驅動3D語音動畫方法依賴於特定角色的訓練資料或繁重的綁定/重新網格化，而AnyTalk透過利用在大型影片資料集上訓練的最新影片擴散模型繞過這些限制。我們首先透過角色特定微調技術（CsF）將預訓練的影片擴散模型適應到目標角色。藉由在3D角色的渲染影像與歸零的音訊嵌入（表示「無動作」）配對上進行微調，我們消除了對動畫資料的需求，同時保留了大型影片擴散模型的動作先驗。接著，我們透過所提出的最佳化過程估計混合變形（blendshape）參數，將生成的說話頭部影片提升為3D語音動畫。AnyTalk能夠在各種面部網格與混合變形配置上實現口型同步動畫，大幅減少手動工作量與資料需求。我們進一步將AnyTalk蒸餾為精簡網路AnyTalk_{RT}，從而實現即時效能，提升其可用性。透過利用說話頭部影片生成技術，我們的方法擴大了任意角色對音訊驅動語音動畫技術的可及性。程式碼已公開於 https://serin-yoon.github.io/projects/anytalk/。

English

We present AnyTalk, a novel method for generating 3D speech animations for arbitrary characters without requiring any animation data. While existing audio-driven 3D speech animation methods rely on character-specific training data or laborious rigging/re-meshing, AnyTalk circumvents these limitations by leveraging recent video diffusion models trained on extensive video datasets. We first adapt a pre-trained video diffusion model to a target character through our Character-specific Fine-tuning (CsF) technique. By fine-tuning on rendered images of the 3D character paired with zeroed-out audio embeddings (representing "no motion"), we eliminate the need for animation data while preserving the motion prior of large-scale video diffusion model. We then uplift the resulting talking-head video into a 3D speech animation by estimating blendshape parameters through a proposed optimization process. AnyTalk enables lip-synced animations across diverse face meshes and blendshape configurations, significantly reducing manual effort and data requirements. We further enhance usability by distilling AnyTalk into a streamlined network, AnyTalk_{RT}, thereby enabling real-time performance. By leveraging talking-head video generation, our method broadens access to audio-driven speech animation technology for arbitrary characters. The code is publicly available at https://serin-yoon.github.io/projects/anytalk/.