讓RGB成為視覺的語言

摘要

本研究提出了一種統一的視覺模型公式，其中將遮罩、深度圖及其他結構化視覺訊號等非自然圖像的多樣視覺資訊，皆表示為RGB影像，而一般視覺任務則可轉換為通用的RGB到RGB影像編輯問題。在此範式中，不同類型的視覺資訊在內部與自然影像共享相同的編碼與解碼架構及參數，使單一模型能透過統一的視覺介面跨任務遷移，此方式類似於語言模型對文字的運作機制。我們將此公式稱為RGB輸入與RGB輸出（RINO）。基於無需特定任務微調的通用影像編輯骨幹，RINO在密集理解任務（如分割與深度估計，我們將輸出統一為RGB）以及密集條件生成任務（如姿態到影像生成，我們將輸入統一為RGB）中，展現出穩健且具競爭力的零樣本效能。期望本研究能為通用統一的視覺語言系統提供實用見解，使多樣視覺任務可透過共享的視覺語言表達、解讀與解決。程式碼已公開於 https://github.com/yangtiming/RINO。

English

This work introduces a unified formulation for vision models, where diverse forms of visual information beyond natural images, such as masks, depth maps, and other structured visual signals, are all represented as RGB images, while general visual tasks can be converted into a common RGB-to-RGB image editing problem. In this paradigm, different types of visual information internally share the same encoding and decoding architecture and parameters as natural images, enabling a single model to transfer across tasks through a unified visual interface, in a way analogous to how language models operate over text. We refer to this formulation as RGB In and RGB Out (RINO). Built upon a generic image editing backbone without task-specific fine-tuning, RINO demonstrates robust and competitive zero-shot performance on both dense understanding tasks such as segmentation and depth estimation (where we unify outputs as RGB), and dense-conditioned generation tasks such as pose-to-image generation (where we unify inputs as RGB). We hope this study provides useful insights toward general unified vision-language systems, where diverse visual tasks can be expressed, interpreted, and solved through a shared visual language. Code is available at https://github.com/yangtiming/RINO.