无瓶颈统一多模态模型的表示强制

摘要

统一多模态模型（UMMs）旨在将感知与生成整合到单一模型中。然而，现有的UMMs仍依赖一个冻结的、单独预训练的VAE进行图像生成，这造成了结构瓶颈。若简单移除VAE，则会引入质量差距，因为模型必须从原始像素中同时学习高层结构和底层细节。本文提出“表示强制”（Representation Forcing, RF）技术，通过将表示预测变为模型的原生能力来弥合这一差距。具体而言，RF迫使解码器在像素之前以自回归方式预测视觉表示作为中间标记；这些标记随后保留在上下文中，并在同一骨干网络内引导像素扩散。通过将表示从感知输出转变为生成目标，RF消除了对外部生成隐空间的依赖。我们发现RF同时有益于理解与生成任务。在图像生成方面，采用RF的像素空间模型达到了最先进的基于VAE的统一模型的水平。在图像理解方面，像素空间RF通常优于其基于VAE的变体。这些结果共同为构建端到端、无瓶颈的UMMs迈出了有效一步。

English

Unified multimodal models (UMMs) aim to handle perception and generation in a single model. Yet existing UMMs still rely on a frozen, separately pretrained VAE for image generation, imposing a structural bottleneck. Naively removing it introduces a quality gap, as the model must learn both high-level structure and low-level details from raw pixels. In this paper, we propose Representation Forcing (RF), a technique that closes this gap by making representation prediction a native capability of the model. Concretely, RF forces the decoder to autoregressively predict visual representations as intermediate tokens before pixels; these tokens then stay in context to guide pixel diffusion within the same backbone. By turning representations from perception outputs into generation targets, RF eliminates the need for any external generative latent space. We find that RF benefits both understanding and generation. On image generation, our pixel-space model with RF matches state-of-the-art VAE-based unified models. On image understanding, pixel-space RF generally outperforms its VAE-based variant. Together, these results offer an effective step toward end-to-end, bottleneck-free UMMs.