GEAR：引導式端到端自迴歸用於影像合成

摘要

视觉生成模型通常分两阶段训练：首先训练一个分词器用于重建，随后将其冻结，再训练一个生成器处理其离散索引或连续潜变量。这种解耦方式导致分词器无法感知生成器易于建模的内容。我们提出GEAR（引导式端到端自回归），通过表征对齐引导，联合端到端训练向量量化（VQ）分词器和自回归（AR）生成器。关键障碍在于，输入AR模型的VQ索引不可微，梯度无法传递至分词器，且直通估计器会失效。GEAR通过码本分配的双重读出机制解决此问题：硬性单热分支训练AR进行下一令牌预测，而可微软性分支则引入表征对齐损失，该损失回流仅引导分词器。由此，AR模型引导其分词器朝向更易预测的索引分布，将对齐负担从分词器转移至AR：分词器自身特征变得不那么像DINOv2，而AR特征则更趋近于此——这与扩散方法中使潜变量本身语义化的策略相反。相比强基线LlamaGen-REPA，GEAR在ImageNet上实现gFID收敛速度提升高达10倍，学习到显著更优的块级及空间一致特征，并泛化至多种量化器（VQVAE、LFQ、IBQ）及文本到图像生成任务。

English

Visual generative models are typically trained in two stages. A tokenizer is first trained for reconstruction and then frozen, after which a generator is trained on its discrete indices or continuous latents. This decoupling leaves the tokenizer unaware of what the generator finds easy to model. We present GEAR (Guided End-to-end AutoRegression), which trains a vector-quantized (VQ) tokenizer and an autoregressive (AR) generator jointly and end-to-end, guided by representation alignment. The key obstacle is that the VQ index fed to the AR model is non-differentiable, so gradients cannot reach the tokenizer, and a straight-through estimator collapses. GEAR resolves this with a dual read-out of the codebook assignment. A hard, one-hot branch trains the AR with next-token prediction, while a differentiable soft branch carries a representation-alignment loss that flows back to guide only the tokenizer. The AR model thereby steers its tokenizer toward an index distribution it can predict more easily. This shifts the alignment burden from the tokenizer to the AR: the tokenizer's own features become less DINOv2-like while the AR's become more so, the opposite of diffusion-side recipes that make the latent itself semantic. GEAR speeds up ImageNet gFID convergence by up to 10x relative to the strong LlamaGen-REPA baseline, learns markedly better patch-level and spatially-coherent features, and generalizes across quantizers (VQVAE, LFQ, IBQ) and to text-to-image generation.