視覺作為統一多模態生成

摘要

我們將電腦視覺表述為統一的多模態生成任務，將異質的視覺任務表達在統一多模態模型的原生文字與影像生成空間中，而無需任務專屬的架構。在此表述下，SenseNova-Vision 使用自然語言指令與可選的視覺提示來指定任務、目標區域或視角，以及解碼慣例，並以文字生成符號輸出、以影像生成密集空間預測、或以文字與影像混合輸出處理組合性任務。為支援大規模訓練，我們將多樣的電腦視覺標註轉換為與這些生成空間相容的指令-回應範例，從而建立 SenseNova-Vision 語料庫——一個涵蓋文字、影像與混合目標的電腦視覺指令-回應語料庫。SenseNova-Vision 從現成的預訓練統一多模態模型出發，主要在此語料庫上進行訓練，並輔以多模態資料作為能力保留的混合資料，且無需任務專屬的預測頭或架構修改。最終的模型涵蓋廣泛的視覺任務，包括偵測、OCR、關鍵點估計、分割、深度估計、表面法線預測、點雲與相機姿態估計，同時支援由語言定義的變體，這些變體結合了類別、顏色、區域及其他視覺線索。實驗顯示，單一統一模型能在結構化視覺理解、密集幾何預測、分割與多視角視覺幾何方面，與領先的任務專屬系統匹敵。這些結果表明統一多模態生成是將電腦視覺能力整合至通用基礎模型的一條可擴展路徑。該模型與語料庫已公開提供。

English

We formulate computer vision as unified multimodal generation, where heterogeneous visual tasks are expressed in the native text and image generation spaces of a unified multimodal model, without task-specific architectures. Under this formulation, SenseNova-Vision uses natural-language instructions and optional visual prompts to specify tasks, target regions or views, and decoding conventions, and generates responses as text for symbolic outputs, images for dense spatial predictions, or mixed text-and-image outputs for compositional tasks. To support large-scale training, we convert diverse computer vision annotations into instruction-response examples compatible with these generation spaces, resulting in the SenseNova-Vision Corpus, a computer-vision instruction-response corpus spanning text, image, and mixed targets. Starting from an off-the-shelf pretrained unified multimodal model, SenseNova-Vision is trained primarily on this corpus, with auxiliary multimodal data used as a capability-preserving mixture, and requires no task-specific prediction heads or architectural modifications. The resulting model covers a broad range of vision tasks, including detection, OCR, keypoint estimation, segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation, while supporting language-defined variants that combine category, color, region, and other visual cues. Experiments show that a single unified model can match leading task-specialized systems across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. These results suggest unified multimodal generation as a scalable route for integrating computer vision capabilities into general-purpose foundation models. The model and corpus are publicly available.