AniPortrait：音频驱动的逼真肖像动画合成

AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation

March 26, 2024

作者: Huawei Wei, Zejun Yang, Zhisheng Wang

cs.AI

摘要

在这项研究中，我们提出了AniPortrait，这是一个新颖的框架，用于生成由音频和参考肖像图像驱动的高质量动画。我们的方法论分为两个阶段。首先，我们从音频中提取3D中间表示，并将其投影到一系列2D面部标记中。随后，我们采用强大的扩散模型，结合运动模块，将标记序列转换为逼真且在时间上连贯的肖像动画。实验结果表明，AniPortrait在面部自然性、姿势多样性和视觉质量方面表现优越，从而提供了增强的感知体验。此外，我们的方法在灵活性和可控性方面展现出相当大的潜力，可以有效应用于面部运动编辑或面部再现等领域。我们在https://github.com/scutzzj/AniPortrait 上发布了代码和模型权重。

English

In this study, we propose AniPortrait, a novel framework for generating high-quality animation driven by audio and a reference portrait image. Our methodology is divided into two stages. Initially, we extract 3D intermediate representations from audio and project them into a sequence of 2D facial landmarks. Subsequently, we employ a robust diffusion model, coupled with a motion module, to convert the landmark sequence into photorealistic and temporally consistent portrait animation. Experimental results demonstrate the superiority of AniPortrait in terms of facial naturalness, pose diversity, and visual quality, thereby offering an enhanced perceptual experience. Moreover, our methodology exhibits considerable potential in terms of flexibility and controllability, which can be effectively applied in areas such as facial motion editing or face reenactment. We release code and model weights at https://github.com/scutzzj/AniPortrait