V-RAE：反思視頻生成中的潛在空間

摘要

潛在影片生成依賴於自編碼器定義一個緊湊的空間，生成模型在其中運行。儘管影片自編碼器架構已大幅演化，但其潛在空間仍主要針對像素級重建進行優化，因而提供的高層語義組織有限。然而，重建最優的潛在空間未必適合生成建模。我們提出 V-RAE，一種影片表徵自編碼器，它在凍結的視覺基礎模型表徵之上構建緊湊的生成潛在表示。一個輕量的時間池化模組在保留語義結構的同時移除時間冗餘，影片解碼器則從壓縮特徵中重建連續運動。我們使用四種具代表性的凍結編碼器，在影片重建、語義探測和類別條件生成上評估 V-RAE。V-RAE 在 K600 上達到 2.13 的 rFVD，優於所有評估過的大規模預訓練影片 VAE。其潛在表示比傳統影片 tokenizer 的潛在表示保留了顯著更多的語義資訊。在匹配的生成設定下，我們的最佳變體在 UCF101 和 K600 上分別達到 117.86 和 19.16 的 gFVD 分數，同時收斂速度最高提升 6 倍。我們進一步證明，僅靠重建品質不足以表徵生成效用，並引入 tFVD——一種與下游生成品質相關性更可靠的時間一致性診斷指標。除了影片生成之外，在匹配的預測設定下，V-RAE 在 Cityscapes 上的未來影片預測也優於 Wan 2.2 VAE 潛在空間。綜上所述，實驗表明凍結的語義表徵能夠支持影片重建、生成和預測建模。專案頁面：https://v-rae.github.io/。

English

Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We propose V-RAE, a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features. We evaluate V-RAE with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation. V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. Under matched generation settings, our best variant achieves gFVD scores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster}. We further show that reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Beyond video generation, V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, the experiments show that frozen semantic representations can support video reconstruction, generation, and predictive modeling. The project page: https://v-rae.github.io/.