唇部强制：面向实时唇音同步的少步自回归扩散

摘要

基于扩散的口型同步模型在视觉质量和音画对齐方面表现出色，但全序列双向注意力机制和大量去噪步骤使其难以实现实时推理。我们提出Lip Forcing——据我们所知，这是首个用于视频到视频（V2V）口型同步的自回归扩散方法，它将一个140亿参数的音频条件双向视频扩散教师模型蒸馏为因果学生模型。推理时，学生模型仅需两步去噪即可生成每个片段，且无需运行时CFG，从而实现实时口型同步。一项针对口型同步的教师轨迹分析揭示了CFG的保真度-同步性权衡：无CFG预测偏向参考保真度，而CFG引导预测则偏向中间轨迹波段内的同步性。Lip Forcing将这一发现转化为三个基于分析的组件：同步窗口DMD、两步推理调度以及基于SyncNet的奖励函数。我们在两个学生模型规模上验证了Lip Forcing，两者均从140亿参数的教师模型蒸馏而来。13亿参数的学生模型在31 FPS下实现实时流式处理，比同等规模的双向模型快17.6倍；140亿参数的学生模型（据报告是目前V2V口型同步最大的扩散模型）在保持可比参考保真度的同时，运行速度比教师模型快39.8倍。两个规模的首帧延迟均低于毫秒级，远低于所有扩散基线方法。

English

Diffusion-based lip synchronization models achieve strong visual quality and audio-visual alignment, but full-sequence bidirectional attention and many denoising steps make them impractical for real-time inference. We present Lip Forcing, to our knowledge the first autoregressive diffusion method for video-to-video (V2V) lip synchronization, which distills a 14B audio-conditioned bidirectional video diffusion teacher into causal students. At inference, the students generate each chunk in only two denoising steps without inference-time CFG, enabling real-time lip synchronization. A lip-sync-specific teacher-trajectory analysis reveals a CFG fidelity-sync tradeoff: no-CFG predictions favor reference fidelity, whereas CFG-guided predictions favor synchronization within a mid-trajectory band. Lip Forcing translates this finding into three analysis-derived components: Sync-Window DMD, a two-step inference schedule, and a SyncNet-based reward. We validate Lip Forcing at two student scales, both distilled from the 14B teacher. The 1.3B student crosses into real-time streaming at 31 FPS, 17.6times faster than its same-scale bidirectional model. The 14B student, the largest diffusion model reported for V2V lip synchronization, runs 39.8times faster than its teacher at comparable reference fidelity. Time-to-first-frame is sub-millisecond at both scales, far below every diffusion baseline.