SkyReels-Audio：全音频条件化视频对话肖像的扩散变换器

摘要

基于多模态输入（包括文本、图像和视频）引导的音频条件说话肖像生成与编辑领域仍待深入探索。本文提出了SkyReels-Audio，一个用于合成高保真且时序连贯的说话肖像视频的统一框架。该框架建立在预训练的视频扩散变换器基础之上，支持无限长度的生成与编辑，同时通过多模态输入实现多样化和可控的条件引导。我们采用混合课程学习策略，逐步对齐音频与面部动作，从而实现对长视频序列的细粒度多模态控制。为了增强局部面部一致性，我们引入了面部掩码损失和音频引导的无分类器指导机制。滑动窗口去噪方法进一步融合了跨时间段的潜在表示，确保了在长时间跨度和多样身份下的视觉保真度和时序一致性。更重要的是，我们构建了一个专门的数据管道，用于筛选高质量的音频、视频及文本描述同步三元组。全面的基准评估表明，SkyReels-Audio在唇形同步准确性、身份一致性以及真实面部动态表现上均取得了卓越性能，尤其在复杂和挑战性条件下表现尤为突出。

English

The generation and editing of audio-conditioned talking portraits guided by multimodal inputs, including text, images, and videos, remains under explored. In this paper, we present SkyReels-Audio, a unified framework for synthesizing high-fidelity and temporally coherent talking portrait videos. Built upon pretrained video diffusion transformers, our framework supports infinite-length generation and editing, while enabling diverse and controllable conditioning through multimodal inputs. We employ a hybrid curriculum learning strategy to progressively align audio with facial motion, enabling fine-grained multimodal control over long video sequences. To enhance local facial coherence, we introduce a facial mask loss and an audio-guided classifier-free guidance mechanism. A sliding-window denoising approach further fuses latent representations across temporal segments, ensuring visual fidelity and temporal consistency across extended durations and diverse identities. More importantly, we construct a dedicated data pipeline for curating high-quality triplets consisting of synchronized audio, video, and textual descriptions. Comprehensive benchmark evaluations show that SkyReels-Audio achieves superior performance in lip-sync accuracy, identity consistency, and realistic facial dynamics, particularly under complex and challenging conditions.

SkyReels-Audio：全音频条件化视频对话肖像的扩散变换器

SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers

摘要

Support