EMO：情感头像生成——在弱条件下利用音频到视频扩散模型生成富有表现力的头像视频

摘要

在这项工作中，我们致力于增强说唱视频生成中的逼真度和表现力，重点关注音频提示和面部运动之间的动态微妙关系。我们确定了传统技术的局限性，通常无法捕捉到完整的人类表情谱系和个体面部风格的独特性。为了解决这些问题，我们提出了EMO，这是一个新颖的框架，采用直接的音频到视频合成方法，绕过了中间的3D模型或面部标志的需要。我们的方法确保了帧之间的无缝过渡和视频中一致的身份保留，从而产生高度表现力和栩栩如生的动画。实验结果表明，EMO不仅能够生成令人信服的说唱视频，还能以各种风格生成歌唱视频，在表现力和逼真度方面明显优于现有的最先进方法。

English

In this work, we tackle the challenge of enhancing the realism and expressiveness in talking head video generation by focusing on the dynamic and nuanced relationship between audio cues and facial movements. We identify the limitations of traditional techniques that often fail to capture the full spectrum of human expressions and the uniqueness of individual facial styles. To address these issues, we propose EMO, a novel framework that utilizes a direct audio-to-video synthesis approach, bypassing the need for intermediate 3D models or facial landmarks. Our method ensures seamless frame transitions and consistent identity preservation throughout the video, resulting in highly expressive and lifelike animations. Experimental results demonsrate that EMO is able to produce not only convincing speaking videos but also singing videos in various styles, significantly outperforming existing state-of-the-art methodologies in terms of expressiveness and realism.

EMO：情感头像生成——在弱条件下利用音频到视频扩散模型生成富有表现力的头像视频

EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions

摘要

Support