GenFirst：先生成后重建的稳定端到端潜空间生成建模

GenFirst: Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling

August 29, 2026

作者: Guangting Zheng, Yiyuan Zhang, Tao Yang, Yunpeng Chen, Rui Zhu, Jiajun Deng, Yanyong Zhang

cs.AI

摘要

潜在生成模型通常采用两阶段流程，先训练变分自编码器进行重建，然后在冻结的潜在空间上训练生成模型。由于针对重建优化的潜在表示不一定有利于生成，联合训练两个模型是一种有吸引力的替代方案。然而，直接端到端训练仍然具有挑战性，因为它容易发生潜在坍缩，并面临生成-重建冲突。我们通过分析不同目标如何塑造潜在空间来重新审视这一问题，并提出了两个关键见解。首先，KL散度目标中的熵项对于防止坍缩至关重要：重建和先验拟合倾向于收缩后验，而熵则保持非退化的潜在不确定性。其次，重建和生成表现出不对称的学习动态：重建速度快且受到强监督，而生成则更慢且更难优化。基于这些见解，我们实现了首次无需潜在坍缩的直接端到端训练，并提出了GenFirst，一种简单的先生成后重建策略。生成目标首先在弱重建压力下塑造潜在空间，然后逐步加强重建以恢复视觉细节。我们使用具有精确似然的连续自回归先验和具有隐式似然的SiT先验来验证GenFirst。借助我们的端到端目标和GenFirst，SiT在ImageNet-256上实现了有CFG时gFID为0.97、无CFG时1.45的成绩，而MMDiT在文本到图像生成任务上达到了GenEval得分0.90。除了图像生成之外，我们将该框架扩展到用于生成和表征学习的共享视觉潜在表示，以及连续的统一文本-图像生成。这些结果证明了稳定的端到端潜在学习跨不同生成先验和模态的通用性。

English

Latent generative models typically follow a two-stage pipeline, training a variational autoencoder for reconstruction and then a generative model on the frozen latent space. Since reconstruction-optimized latents are not necessarily generation-friendly, jointly training both models is an appealing alternative. However, direct end-to-end training remains challenging, as it is prone to latent collapse and faces a generation-reconstruction conflict. We revisit this problem by analyzing how different objectives shape the latent space and identify two key insights. First, the entropy term in the Kullback-Leibler divergence objective is essential for preventing collapse: reconstruction and prior fitting tend to shrink the posterior, while entropy preserves non-degenerate latent uncertainty. Second, reconstruction and generation exhibit asymmetric learning dynamics: reconstruction is fast and strongly supervised, whereas generation is slower and harder to optimize. Based on these insights, we achieve the first direct end-to-end training without latent collapse and propose GenFirst, a simple generation-before-reconstruction strategy. The generative objective first shapes the latent space under weak reconstruction pressure, after which reconstruction is progressively strengthened to recover visual details. We validate GenFirst with continuous autoregressive priors with exact likelihoods and SiT priors with implicit likelihoods. With our end-to-end objective and GenFirst, SiT achieves a gFID of 0.97 with CFG and 1.45 without CFG on ImageNet-256, while MMDiT reaches a GenEval score of 0.90 on text-to-image generation. Beyond image generation, we extend the framework to shared visual latents for generation and representation learning, and to continuous unified text-image generation. These results demonstrate the generality of stable end-to-end latent learning across generative priors and modalities.