V-RAE：重新思考用于生成的视频潜空间

摘要

潜在视频生成依赖于自编码器来定义一个紧凑的空间，生成模型在其中运行。尽管视频自编码器架构已有显著发展，但其潜在空间仍主要针对像素级重建进行优化，所能提供的高层语义组织有限。然而，重建最优的潜在空间未必适合生成建模。我们提出V-RAE，一种视频表征自编码器，在冻结的视觉基础模型表征之上构建紧凑的生成潜在表示。一个轻量级时间池化模块在保留语义结构的同时去除时间冗余，视频解码器则从压缩特征中重建连续运动。我们使用四种代表性的冻结编码器评估V-RAE的视频重建、语义探测和类别条件生成性能。V-RAE在K600上取得2.13的rFVD，优于所有被评估的大规模预训练视频VAE。其潜在表示比传统视频分词器的潜在表示保留了明显更多的语义信息。在匹配的生成设置下，我们的最佳变体在UCF101和K600上分别取得117.86和19.16的gFVD分数，同时收敛速度最高提升6倍。我们进一步表明，仅靠重建质量不足以刻画生成效用，并引入tFVD，一种与下游生成质量相关性更可靠的时间一致性诊断指标。在视频生成之外，在匹配的预测设置下，与Wan 2.2 VAE潜在空间相比，V-RAE还提升了Cityscapes上的未来视频预测性能。综合来看，这些实验表明，冻结的语义表征能够支持视频重建、生成和预测建模。项目页面：https://v-rae.github.io/。

English

Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We propose V-RAE, a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features. We evaluate V-RAE with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation. V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. Under matched generation settings, our best variant achieves gFVD scores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster}. We further show that reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Beyond video generation, V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, the experiments show that frozen semantic representations can support video reconstruction, generation, and predictive modeling. The project page: https://v-rae.github.io/.