GenFirst：以先生成後重建實現穩定的端到端潛在生成建模

GenFirst: Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling

August 29, 2026

作者: Guangting Zheng, Yiyuan Zhang, Tao Yang, Yunpeng Chen, Rui Zhu, Jiajun Deng, Yanyong Zhang

cs.AI

摘要

潛在生成模型通常遵循兩階段流程：先訓練變分自編碼器進行重建，再在凍結的潛在空間上訓練生成模型。由於以重建優化的潛在表徵未必利於生成，聯合訓練兩個模型便成為一個具吸引力的替代方案。然而，直接的端到端訓練仍具挑戰性，因為它容易發生潛在崩潰，並面臨生成與重建之間的衝突。我們重新審視此問題，分析不同目標函數如何塑造潛在空間，並提出兩項關鍵洞見。首先，Kullback-Leibler 散度目標中的熵項對防止崩潰至關重要：重建與先驗擬合傾向於收縮後驗，而熵則保留非退化的潛在不確定性。其次，重建與生成表現出不對稱的學習動態：重建快速且具有強監督，而生成則較慢且更難優化。基於這些洞見，我們首次實現了無潛在崩潰的直接端到端訓練，並提出 GenFirst——一種簡單的「先生成、後重建」策略。生成目標先在較弱的重建壓力下塑造潛在空間，隨後逐步加強重建以恢復視覺細節。我們使用具有精確似然的連續自迴歸先驗以及具有隱式似然的 SiT 先驗來驗證 GenFirst。憑藉我們的端到端目標與 GenFirst，SiT 在 ImageNet-256 上達到有 CFG 時 gFID 0.97、無 CFG 時 1.45 的表現，而 MMDiT 在文字到圖像生成任務上達到 GenEval 0.90 的分數。除圖像生成外，我們將此框架擴展至用於生成與表徵學習的共享視覺潛在表徵，以及連續的統一文字-圖像生成。這些結果證明了穩定端到端潛在學習在各種生成先驗與模態間具有普遍適用性。

English

Latent generative models typically follow a two-stage pipeline, training a variational autoencoder for reconstruction and then a generative model on the frozen latent space. Since reconstruction-optimized latents are not necessarily generation-friendly, jointly training both models is an appealing alternative. However, direct end-to-end training remains challenging, as it is prone to latent collapse and faces a generation-reconstruction conflict. We revisit this problem by analyzing how different objectives shape the latent space and identify two key insights. First, the entropy term in the Kullback-Leibler divergence objective is essential for preventing collapse: reconstruction and prior fitting tend to shrink the posterior, while entropy preserves non-degenerate latent uncertainty. Second, reconstruction and generation exhibit asymmetric learning dynamics: reconstruction is fast and strongly supervised, whereas generation is slower and harder to optimize. Based on these insights, we achieve the first direct end-to-end training without latent collapse and propose GenFirst, a simple generation-before-reconstruction strategy. The generative objective first shapes the latent space under weak reconstruction pressure, after which reconstruction is progressively strengthened to recover visual details. We validate GenFirst with continuous autoregressive priors with exact likelihoods and SiT priors with implicit likelihoods. With our end-to-end objective and GenFirst, SiT achieves a gFID of 0.97 with CFG and 1.45 without CFG on ImageNet-256, while MMDiT reaches a GenEval score of 0.90 on text-to-image generation. Beyond image generation, we extend the framework to shared visual latents for generation and representation learning, and to continuous unified text-image generation. These results demonstrate the generality of stable end-to-end latent learning across generative priors and modalities.