VideoRAE：通过表示自编码器驾驭视频基础模型进行生成建模

摘要

視頻生成模型通常依賴於由3D變分自編碼器（3D-VAE）學習得到的潛在空間。然而，傳統的3D-VAE主要針對像素級重建進行優化，這可能限制了其潛在變量所捕捉的語義與時空結構。與此同時，V-JEPA 2和VideoMAEv2等視頻基礎模型（VFM）展現出強大的視頻理解能力，但其凍結表徵能否轉化為緊湊、可重建且利於生成的視頻潛在變量，仍有待探索。針對此問題，我們提出VideoRAE，這是一種表徵自編碼器，利用來自凍結視頻基礎編碼器的多尺度分層特徵，並透過輕量級1D自注意力投影器進行壓縮。VideoRAE支援用於擴散變換器的連續潛在變量，以及透過多碼書高維量化用於自迴歸模型的離散token。在解碼階段，與凍結VFM教師模型的局部與全域表徵對齊目標，可提升語義保留能力，並實現無需KL正則化的訓練。實驗顯示，VideoRAE在連續與離散模式下均能實現強勁的重建效果。在UCF-101上，分別以AR和DiT生成器獲得40和93的類別到視頻gFVD最新最佳結果，同時收斂速度比競爭的自編碼器基準快約5倍。在一項受控的2B規模文字轉視頻研究中，用VideoRAE取代LTX-VAE可在相近設定下實現更快的收斂。這些結果驗證了凍結VFM表徵作為通用且利於生成的視頻潛在變量的有效性。模型與程式碼將於https://zhxie0117.github.io/VideoRAE發布。

English

Video generative models commonly rely on latent spaces learned by 3D Variational Autoencoders (3D-VAEs). However, conventional 3D-VAEs are mainly optimized for pixel-level reconstruction, which can limit the semantic and spatio-temporal structure captured by their latents. Meanwhile, Video Foundation Models (VFMs) such as V-JEPA 2 and VideoMAEv2 show strong video understanding capabilities, yet whether their frozen representations can be transformed into compact, reconstruction-capable, and generation-friendly video latents remains largely unexplored. We answer this question with VideoRAE, a representation autoencoder that leverages multi-scale hierarchical features from a frozen video foundation encoder and compresses them with a lightweight 1D self-attention projector. VideoRAE supports both continuous latents for Diffusion Transformers and discrete tokens for autoregressive models via multi-codebook high-dimensional quantization. During decoding, a local-and-global representation alignment objective with the frozen VFM teacher improves semantic preservation and enables training without KL regularization. Experiments show that VideoRAE achieves strong reconstruction in both continuous and discrete regimes. On UCF-101, it obtains state-of-the-art class-to-video gFVDs of 40 and 93 with AR and DiT generators, respectively, while converging approximately 5x faster than competing autoencoder baselines. In a controlled 2B-scale text-to-video study, replacing LTX-VAE with VideoRAE leads to faster convergence under comparable settings. These results validate frozen VFM representations as versatile and generation-friendly video latents. The model and code will be released on https://zhxie0117.github.io/VideoRAE.