視覺語言定位作為雙向概念對應

摘要

視覺-語言定位將語言與視覺內容相連，然而現有的大多數形式化方法將定位簡化為單向定位問題：給定預先指定的文本詞組或類別名稱，識別對應的圖像區域。這種設定假設相關的語言單位已知，從而忽略了接地溝通中一個更基本的挑戰：確定文本的哪些部分具有視覺指稱性，以及它們如何與圖像中的實體相對應。我們將定位形式化為圖像-文本對之間的雙向概念對應。給定一幅圖像及其配對文本，目標是恢復視覺指稱文本片段與實例級圖像分割區段之間的所有對應關係，而無需假設相關文本片段已提供。此形式化將文本分割、圖像分割和跨模態對齊視為單一的對應預測問題，從而統一了常見的定位任務，包括詞組定位、指稱表達定位和開放詞彙偵測。為了解決此任務，我們提出了 ConCor-1，這是一個建立在預訓練視覺-語言模型之上的定位模型。它使用可學習的橋接標記來表示候選的圖像-文本對應關係，並為每個標記預測文本遮罩、圖像遮罩和對應存在分數。為了訓練和評估此任務，我們將多樣化的定位和分割資料集轉換為統一的對應格式。實驗結果表明，ConCor-1 持續優於基線方法，在長字幕資料集上將對應關係 F1 提升了 48%，在零樣本 LVIS 上提升了 29%，其中大型類別列表作為文本輸入。

English

Vision-language grounding connects language to visual content, yet most existing formulations reduce grounding to a unidirectional localization problem: given a prespecified text phrase or category name, identify the corresponding image region. This setup assumes that the relevant linguistic unit is already known, overlooking a more basic challenge in grounded communication: determining which parts of the text are visually referential and how they correspond to entities in the image. We formulate grounding as bidirectional concept correspondence over an image-text pair. Given an image and its paired text, the goal is to recover all correspondences between visually referential text spans and instance-level image segments, without assuming that the relevant text spans are provided. This formulation unifies common grounding tasks, including phrase grounding, referring expression grounding, and open-vocabulary detection, by treating text segmentation, image segmentation, and cross-modal alignment as a single correspondence prediction problem. To address this task, we introduce ConCor-1, a grounding model built on top of a pretrained vision-language model. It uses learnable bridge tokens to represent candidate image-text correspondences and predicts, for each token, a text mask, an image mask, and a correspondence presence score. To train and evaluate this task, we convert diverse grounding and segmentation datasets into a unified correspondence format. Experiments show that ConCor-1 consistently outperforms baselines, improving correspondence F1 by 48% on the long-caption dataset and by 29% on zero-shot LVIS, where the large category list serves as the text input.