自迴歸視頻擴散模型的錯誤分析：一個統一框架

Error Analyses of Auto-Regressive Video Diffusion Models: A Unified Framework

March 12, 2025

作者: Jing Wang, Fengzhuo Zhang, Xiaoli Li, Vincent Y. F. Tan, Tianyu Pang, Chao Du, Aixin Sun, Zhuoran Yang

cs.AI

摘要

多種自回歸視頻擴散模型（ARVDM）在生成逼真的長視頻方面取得了顯著成功。然而，對這些模型的理論分析仍然不足。在本研究中，我們為這些模型建立了理論基礎，並利用我們的見解來提升現有模型的性能。我們首先開發了Meta-ARVDM，這是一個統一框架，涵蓋了大多數現有方法。通過Meta-ARVDM，我們分析了由Meta-ARVDM生成的視頻與真實視頻之間的KL散度。我們的分析揭示了ARVDM固有的兩個重要現象——誤差累積和記憶瓶頸。通過推導信息論上的不可能性結果，我們表明記憶瓶頸現象無法避免。為了緩解記憶瓶頸，我們設計了各種網絡結構，以顯式地利用更多的過去幀。我們還通過壓縮幀，在緩解記憶瓶頸和推理效率之間實現了顯著改善的平衡。在DMLab和Minecraft上的實驗結果驗證了我們方法的有效性。我們的實驗還展示了不同方法在誤差累積和記憶瓶頸之間的帕累托前沿。

English

A variety of Auto-Regressive Video Diffusion Models (ARVDM) have achieved remarkable successes in generating realistic long-form videos. However, theoretical analyses of these models remain scant. In this work, we develop theoretical underpinnings for these models and use our insights to improve the performance of existing models. We first develop Meta-ARVDM, a unified framework of ARVDMs that subsumes most existing methods. Using Meta-ARVDM, we analyze the KL-divergence between the videos generated by Meta-ARVDM and the true videos. Our analysis uncovers two important phenomena inherent to ARVDM -- error accumulation and memory bottleneck. By deriving an information-theoretic impossibility result, we show that the memory bottleneck phenomenon cannot be avoided. To mitigate the memory bottleneck, we design various network structures to explicitly use more past frames. We also achieve a significantly improved trade-off between the mitigation of the memory bottleneck and the inference efficiency by compressing the frames. Experimental results on DMLab and Minecraft validate the efficacy of our methods. Our experiments also demonstrate a Pareto-frontier between the error accumulation and memory bottleneck across different methods.