UniWorld：高分辨率語義編碼器，實現統一視覺理解與生成

UniWorld: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation

June 3, 2025

作者: Bin Lin, Zongjian Li, Xinhua Cheng, Yuwei Niu, Yang Ye, Xianyi He, Shenghai Yuan, Wangbo Yu, Shaodong Wang, Yunyang Ge, Yatian Pang, Li Yuan

cs.AI

摘要

尽管现有的统一模型在视觉语言理解和文本到图像生成方面表现出色，但这些模型在探索图像感知和操作任务方面存在局限，而这些任务正是用户广泛应用的迫切需求。最近，OpenAI发布了其强大的GPT-4o-Image模型，用于全面的图像感知和操作，展现了卓越的表达能力并引起了社区的广泛关注。通过在我们精心设计的实验中观察GPT-4o-Image的表现，我们推断GPT-4o-Image利用了语义编码器提取的特征，而非VAE（变分自编码器），而VAE在许多图像操作模型中被视为核心组件。受此启发，我们提出了一个名为UniWorld的统一生成框架，该框架基于强大的视觉语言模型和对比语义编码器提供的语义特征。结果，我们仅使用BAGEL数据量的1%构建了一个强大的统一模型，在图像编辑基准测试中持续超越BAGEL。UniWorld还保持了竞争力的图像理解和生成能力，在多个图像感知任务中表现出色。我们完全开源了我们的模型，包括模型权重、训练和评估脚本以及数据集。

English

Although existing unified models deliver strong performance on vision-language understanding and text-to-image generation, their models are limited in exploring image perception and manipulation tasks, which are urgently desired by users for wide applications. Recently, OpenAI released their powerful GPT-4o-Image model for comprehensive image perception and manipulation, achieving expressive capability and attracting community interests. By observing the performance of GPT-4o-Image in our carefully constructed experiments, we infer that GPT-4o-Image leverages features extracted by semantic encoders instead of VAE, while VAEs are considered essential components in many image manipulation models. Motivated by such inspiring observations, we present a unified generative framework named UniWorld based on semantic features provided by powerful visual-language models and contrastive semantic encoders. As a result, we build a strong unified model using only 1% amount of BAGEL's data, which consistently outperforms BAGEL on image editing benchmarks. UniWorld also maintains competitive image understanding and generation capabilities, achieving strong performance across multiple image perception tasks. We fully open-source our models, including model weights, training and evaluation scripts, and datasets.