視覺情境提示

摘要

大型語言模型中的上下文提示已成為提升零樣本能力的常用方法，但該思路在視覺領域的探索仍較少。現有視覺提示方法主要聚焦於指涉分割以標註最相關物件，卻難以應對開放式分割與檢測等多種通用視覺任務。本文提出一種適用於這兩類任務的通用視覺上下文提示框架：我們基於編碼器-解碼器架構，開發了支援筆劃、方框、點選等多樣化提示的靈活提示編碼器，並進一步使其能接納任意數量的參考圖像區塊作為上下文。大量實驗表明，所提出的視覺上下文提示能激發卓越的指涉與通用分割能力，在封閉式領域內數據集上達到具競爭力的表現，並在多個開放式分割數據集展現優異效果。通過聯合訓練COCO與SA-1B數據集，我們的模型在COCO上獲得57.7 PQ，在ADE20K上達到23.2 PQ。程式碼將發佈於https://github.com/UX-Decoder/DINOv。

English

In-context prompting in large language models (LLMs) has become a prevalent approach to improve zero-shot capabilities, but this idea is less explored in the vision domain. Existing visual prompting methods focus on referring segmentation to segment the most relevant object, falling short of addressing many generic vision tasks like open-set segmentation and detection. In this paper, we introduce a universal visual in-context prompting framework for both tasks. In particular, we build on top of an encoder-decoder architecture, and develop a versatile prompt encoder to support a variety of prompts like strokes, boxes, and points. We further enhance it to take an arbitrary number of reference image segments as the context. Our extensive explorations show that the proposed visual in-context prompting elicits extraordinary referring and generic segmentation capabilities to refer and detect, yielding competitive performance to close-set in-domain datasets and showing promising results on many open-set segmentation datasets. By joint training on COCO and SA-1B, our model achieves 57.7 PQ on COCO and 23.2 PQ on ADE20K. Code will be available at https://github.com/UX-Decoder/DINOv.

視覺情境提示

Visual In-Context Prompting

摘要

Support