OmniForcing：即時聯合視聽生成技術的全面釋放

摘要

近期聯合音訊-視覺擴散模型雖實現了卓越的生成品質，卻因雙向注意力依賴導致高延遲，阻礙了即時應用。我們提出OmniForcing——首個將離線雙向擴散模型蒸餾為高保真串流自回歸生成器的框架。然而，對此類雙流架構直接應用因果蒸餾會引發嚴重訓練不穩定性，根源在於模態間極端的時間不對稱性及由此產生的令牌稀疏問題。我們通過引入具零截斷全局前綴的非對稱區塊因果對齊機制，解決內在的資訊密度差距，防止多模態同步漂移。針對因果偏移期間因音訊令牌極度稀疏導致的梯度爆炸問題，進一步採用配備恆等旋轉位置編碼約束的音訊匯點令牌機制。最後，通過聯合自強制蒸餾範式，使模型能在長序列推演中動態修正因曝光偏差產生的累積跨模態誤差。憑藉模態無關的滾動鍵值快取推論方案，OmniForcing在單一GPU上實現每秒25幀的頂尖串流生成效能，同時保持與雙向教師模型相當的多模態同步性與視覺品質。專案頁面：https://omniforcing.com

English

Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency due to their bidirectional attention dependencies, hindering real-time applications. We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autoregressive generator. However, naively applying causal distillation to such dual-stream architectures triggers severe training instability, due to the extreme temporal asymmetry between modalities and the resulting token sparsity. We address the inherent information density gap by introducing an Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix that prevents multi-modal synchronization drift. The gradient explosion caused by extreme audio token sparsity during the causal shift is further resolved through an Audio Sink Token mechanism equipped with an Identity RoPE constraint. Finally, a Joint Self-Forcing Distillation paradigm enables the model to dynamically self-correct cumulative cross-modal errors from exposure bias during long rollouts. Empowered by a modality-independent rolling KV-cache inference scheme, OmniForcing achieves state-of-the-art streaming generation at sim25 FPS on a single GPU, maintaining multi-modal synchronization and visual quality on par with the bidirectional teacher.Project Page: https://omniforcing.com{https://omniforcing.com}

OmniForcing：即時聯合視聽生成技術的全面釋放

OmniForcing: Unleashing Real-time Joint Audio-Visual Generation

摘要

Support