视觉即统一多模态生成

摘要

我们将计算机视觉表述为统一的多模态生成，其中异构的视觉任务在统一多模态模型的原生文本与图像生成空间中被表达，无需依赖任务特定的架构。基于这一表述，SenseNova-Vision 使用自然语言指令与可选的视觉提示来指定任务、目标区域或视角以及解码约定，并以文本形式输出符号化结果、以图像形式输出密集空间预测、或以图文混合形式输出组合任务结果。为支持大规模训练，我们将多样化的计算机视觉标注数据转换为与该生成空间兼容的指令-响应示例，从而构建了 SenseNova-Vision 语料库——一个涵盖文本、图像及混合目标的计算机视觉指令-响应数据集。SenseNova-Vision 从现成的预训练统一多模态模型出发，主要在该语料库上进行训练，并辅以多模态数据作为能力保持的混合，无需任务特定的预测头或架构修改。最终模型覆盖广泛的视觉任务，包括检测、OCR、关键点估计、分割、深度估计、表面法线预测、点云图以及相机位姿估计，同时支持基于语言定义的变体，这些变体结合了类别、颜色、区域及其他视觉线索。实验表明，单一统一模型在结构化视觉理解、密集几何预测、分割以及多视角视觉几何方面能够与领先的任务专用系统匹敌。这些结果提示，统一多模态生成是将计算机视觉能力融入通用基础模型的一条可扩展路径。该模型及语料库均已公开。

English

We formulate computer vision as unified multimodal generation, where heterogeneous visual tasks are expressed in the native text and image generation spaces of a unified multimodal model, without task-specific architectures. Under this formulation, SenseNova-Vision uses natural-language instructions and optional visual prompts to specify tasks, target regions or views, and decoding conventions, and generates responses as text for symbolic outputs, images for dense spatial predictions, or mixed text-and-image outputs for compositional tasks. To support large-scale training, we convert diverse computer vision annotations into instruction-response examples compatible with these generation spaces, resulting in the SenseNova-Vision Corpus, a computer-vision instruction-response corpus spanning text, image, and mixed targets. Starting from an off-the-shelf pretrained unified multimodal model, SenseNova-Vision is trained primarily on this corpus, with auxiliary multimodal data used as a capability-preserving mixture, and requires no task-specific prediction heads or architectural modifications. The resulting model covers a broad range of vision tasks, including detection, OCR, keypoint estimation, segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation, while supporting language-defined variants that combine category, color, region, and other visual cues. Experiments show that a single unified model can match leading task-specialized systems across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. These results suggest unified multimodal generation as a scalable route for integrating computer vision capabilities into general-purpose foundation models. The model and corpus are publicly available.