DAWN：具有非自回归扩散框架的动态帧头像，用于生成说话头视频

DAWN: Dynamic Frame Avatar with Non-autoregressive Diffusion Framework for Talking Head Video Generation

October 17, 2024

作者: Hanbo Cheng, Limin Lin, Chenyu Liu, Pengcheng Xia, Pengfei Hu, Jiefeng Ma, Jun Du, Jia Pan

cs.AI

摘要

头部生成旨在从单个肖像和语音音频剪辑中生成生动逼真的说话头部视频。尽管扩散式头部生成取得了显著进展，但几乎所有方法都依赖自回归策略，这些策略存在于当前生成步骤之外有限的上下文利用、误差累积和较慢的生成速度。为了解决这些挑战，我们提出了DAWN（具有非自回归扩散的动态帧头像），这是一个框架，可以实现动态长度视频序列的一次生成。具体而言，它包括两个主要组件：（1）在潜在运动空间中生成由音频驱动的整体面部动态，以及（2）由音频驱动的头部姿势和眨眼生成。大量实验证明，我们的方法生成具有精确嘴唇运动和自然姿势/眨眼动作的真实生动视频。此外，DAWN具有较高的生成速度，并具有强大的外推能力，确保稳定生成高质量的长视频。这些结果突显了DAWN在说话头部视频生成领域的巨大潜力和潜在影响。此外，我们希望DAWN能激发对扩散模型中非自回归方法的进一步探索。我们的代码将在 https://github.com/Hanbo-Cheng/DAWN-pytorch 上公开。

English

Talking head generation intends to produce vivid and realistic talking head videos from a single portrait and speech audio clip. Although significant progress has been made in diffusion-based talking head generation, almost all methods rely on autoregressive strategies, which suffer from limited context utilization beyond the current generation step, error accumulation, and slower generation speed. To address these challenges, we present DAWN (Dynamic frame Avatar With Non-autoregressive diffusion), a framework that enables all-at-once generation of dynamic-length video sequences. Specifically, it consists of two main components: (1) audio-driven holistic facial dynamics generation in the latent motion space, and (2) audio-driven head pose and blink generation. Extensive experiments demonstrate that our method generates authentic and vivid videos with precise lip motions, and natural pose/blink movements. Additionally, with a high generation speed, DAWN possesses strong extrapolation capabilities, ensuring the stable production of high-quality long videos. These results highlight the considerable promise and potential impact of DAWN in the field of talking head video generation. Furthermore, we hope that DAWN sparks further exploration of non-autoregressive approaches in diffusion models. Our code will be publicly at https://github.com/Hanbo-Cheng/DAWN-pytorch.