UniWorld：面向统一视觉理解与生成的高分辨率语义编码器

UniWorld: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation

June 3, 2025

作者: Bin Lin, Zongjian Li, Xinhua Cheng, Yuwei Niu, Yang Ye, Xianyi He, Shenghai Yuan, Wangbo Yu, Shaodong Wang, Yunyang Ge, Yatian Pang, Li Yuan

cs.AI

摘要

尽管现有的统一模型在视觉-语言理解和文本到图像生成方面表现出色，但其在探索图像感知与操控任务上存在局限，而这些任务正是用户广泛应用的迫切需求。近期，OpenAI发布了其强大的GPT-4o-Image模型，用于全面的图像感知与操控，展现了卓越的表达能力并引发了社区的高度关注。通过在我们精心设计的实验中观察GPT-4o-Image的表现，我们推断该模型利用了语义编码器提取的特征而非变分自编码器（VAE），而VAE在许多图像操控模型中被视为核心组件。受此启发，我们提出了一个名为UniWorld的统一生成框架，该框架基于强大的视觉-语言模型和对比语义编码器提供的语义特征。结果表明，我们仅使用BAGEL数据量的1%便构建了一个强大的统一模型，在图像编辑基准测试中持续超越BAGEL。同时，UniWorld在图像理解与生成能力上保持竞争力，在多项图像感知任务中均取得了优异表现。我们全面开源了模型，包括模型权重、训练与评估脚本以及数据集。

English

Although existing unified models deliver strong performance on vision-language understanding and text-to-image generation, their models are limited in exploring image perception and manipulation tasks, which are urgently desired by users for wide applications. Recently, OpenAI released their powerful GPT-4o-Image model for comprehensive image perception and manipulation, achieving expressive capability and attracting community interests. By observing the performance of GPT-4o-Image in our carefully constructed experiments, we infer that GPT-4o-Image leverages features extracted by semantic encoders instead of VAE, while VAEs are considered essential components in many image manipulation models. Motivated by such inspiring observations, we present a unified generative framework named UniWorld based on semantic features provided by powerful visual-language models and contrastive semantic encoders. As a result, we build a strong unified model using only 1% amount of BAGEL's data, which consistently outperforms BAGEL on image editing benchmarks. UniWorld also maintains competitive image understanding and generation capabilities, achieving strong performance across multiple image perception tasks. We fully open-source our models, including model weights, training and evaluation scripts, and datasets.