VideoRAE: 利用表示自编码器驾驭视频基础模型进行生成建模

摘要

视频生成模型通常依赖于由3D变分自编码器（3D-VAE）学习的潜在空间。然而，传统3D-VAE主要针对像素级重建进行优化，这限制了其潜在表示所捕获的语义和时空结构。与此同时，视频基础模型（VFM）如V-JEPA 2和VideoMAEv2展现了强大的视频理解能力，但其冻结表示能否转化为紧凑、可重建且利于生成的视频潜在变量，目前仍基本未得到探索。我们通过VideoRAE回答了这一问题，它是一种表示自编码器，利用来自冻结视频基础编码器的多尺度层次特征，并通过轻量级1D自注意力投影器对其进行压缩。VideoRAE通过多码本高维量化，同时支持用于扩散变换器的连续潜在变量和用于自回归模型的离散标记。在解码过程中，与冻结的VFM教师模型进行局部与全局表示对齐的目标，提升了语义保留能力，并使得训练无需KL正则化。实验表明，VideoRAE在连续和离散模式下均实现了强大的重建性能。在UCF-101上，使用自回归生成器和扩散变换器生成器分别获得了40和93的最优类别到视频gFVD分数，同时收敛速度比竞争对手的自编码器基线快约5倍。在一项受控的2B规模文本到视频研究中，用VideoRAE替换LTX-VAE在可比设置下实现了更快的收敛。这些结果验证了冻结的VFM表示是通用且利于生成的视频潜在变量。模型和代码将在https://zhxie0117.github.io/VideoRAE上发布。

English

Video generative models commonly rely on latent spaces learned by 3D Variational Autoencoders (3D-VAEs). However, conventional 3D-VAEs are mainly optimized for pixel-level reconstruction, which can limit the semantic and spatio-temporal structure captured by their latents. Meanwhile, Video Foundation Models (VFMs) such as V-JEPA 2 and VideoMAEv2 show strong video understanding capabilities, yet whether their frozen representations can be transformed into compact, reconstruction-capable, and generation-friendly video latents remains largely unexplored. We answer this question with VideoRAE, a representation autoencoder that leverages multi-scale hierarchical features from a frozen video foundation encoder and compresses them with a lightweight 1D self-attention projector. VideoRAE supports both continuous latents for Diffusion Transformers and discrete tokens for autoregressive models via multi-codebook high-dimensional quantization. During decoding, a local-and-global representation alignment objective with the frozen VFM teacher improves semantic preservation and enables training without KL regularization. Experiments show that VideoRAE achieves strong reconstruction in both continuous and discrete regimes. On UCF-101, it obtains state-of-the-art class-to-video gFVDs of 40 and 93 with AR and DiT generators, respectively, while converging approximately 5x faster than competing autoencoder baselines. In a controlled 2B-scale text-to-video study, replacing LTX-VAE with VideoRAE leads to faster convergence under comparable settings. These results validate frozen VFM representations as versatile and generation-friendly video latents. The model and code will be released on https://zhxie0117.github.io/VideoRAE.