無瓶頸統一多模態模型的表徵強制

摘要

统一多模态模型旨在以单一模型处理感知与生成任务。然而现有的统一多模态模型仍依赖冻结的、独立预训练的变分自编码器（VAE）进行图像生成，这构成了结构瓶颈。若直接移除该模块，模型需从原始像素中同时学习高层结构与低层细节，从而产生质量差距。本文提出表征强制（RF）技术，通过将表征预测内化为模型原生能力来弥合这一差距。具体而言，RF强制解码器在生成像素之前，以自回归方式将视觉表征作为中间令牌进行预测；随后这些令牌保留在上下文中，引导同一主干网络内的像素扩散过程。通过将表征从感知输出转变为生成目标，RF消除了对外部生成潜在空间的依赖。我们发现RF对理解与生成任务均有助益。在图像生成方面，采用RF的像素空间模型达到了基于VAE的最先进统一模型的性能水平。在图像理解方面，像素空间RF普遍优于基于VAE的变体。这些结果共同为构建无瓶颈的端到端统一多模态模型迈出了有效一步。

English

Unified multimodal models (UMMs) aim to handle perception and generation in a single model. Yet existing UMMs still rely on a frozen, separately pretrained VAE for image generation, imposing a structural bottleneck. Naively removing it introduces a quality gap, as the model must learn both high-level structure and low-level details from raw pixels. In this paper, we propose Representation Forcing (RF), a technique that closes this gap by making representation prediction a native capability of the model. Concretely, RF forces the decoder to autoregressively predict visual representations as intermediate tokens before pixels; these tokens then stay in context to guide pixel diffusion within the same backbone. By turning representations from perception outputs into generation targets, RF eliminates the need for any external generative latent space. We find that RF benefits both understanding and generation. On image generation, our pixel-space model with RF matches state-of-the-art VAE-based unified models. On image understanding, pixel-space RF generally outperforms its VAE-based variant. Together, these results offer an effective step toward end-to-end, bottleneck-free UMMs.