让RGB成为视觉的语言

摘要

本工作提出了一种视觉模型的统一框架，其中除自然图像外的多种视觉信息形式（如掩码、深度图及其他结构化视觉信号）均可表示为RGB图像，而通用视觉任务则可转化为一个标准的RGB到RGB图像编辑问题。在此范式下，不同类型的视觉信息在内部与自然图像共享相同的编码解码架构及参数，从而使单一模型能够通过统一的视觉接口跨任务迁移，其运作方式类似于语言模型对文本的处理。我们将这一框架称为RGB输入与RGB输出（RINO）。基于通用图像编辑骨干网络且无需任务特定微调，RINO在密集理解任务（如分割与深度估计，我们将输出统一为RGB）和密集条件生成任务（如姿态到图像生成，我们将输入统一为RGB）上均展现出稳健且具有竞争力的零样本性能。我们期望这项研究能为通用统一视觉-语言系统提供有益启示——在该系统中，多样化的视觉任务可通过共享的视觉语言进行表达、解释与求解。代码已开源：https://github.com/yangtiming/RINO。

English

This work introduces a unified formulation for vision models, where diverse forms of visual information beyond natural images, such as masks, depth maps, and other structured visual signals, are all represented as RGB images, while general visual tasks can be converted into a common RGB-to-RGB image editing problem. In this paradigm, different types of visual information internally share the same encoding and decoding architecture and parameters as natural images, enabling a single model to transfer across tasks through a unified visual interface, in a way analogous to how language models operate over text. We refer to this formulation as RGB In and RGB Out (RINO). Built upon a generic image editing backbone without task-specific fine-tuning, RINO demonstrates robust and competitive zero-shot performance on both dense understanding tasks such as segmentation and depth estimation (where we unify outputs as RGB), and dense-conditioned generation tasks such as pose-to-image generation (where we unify inputs as RGB). We hope this study provides useful insights toward general unified vision-language systems, where diverse visual tasks can be expressed, interpreted, and solved through a shared visual language. Code is available at https://github.com/yangtiming/RINO.