所有視覺標記都同等重要嗎？用於視覺語言檢索的保留物件證據的標記合併方法

摘要

多向量視覺語言檢索通過最大相似度後期交互保留了細粒度的視覺證據，但密集的圖像端令牌使得存儲和評分成本高昂。現有的令牌壓縮方法降低了這一成本，但它們可能移除或壓縮了未來查詢令牌可能需要選擇的物件級別和區域級別證據。我們提出SaMer，一個物體感知令牌合併框架，將圖像端投影後的令牌壓縮成K個代表性質心，同時保留原始的後期交互介面。SaMer僅在訓練期間使用物體註釋作為合併先驗，以抑制跨實例混合，在推理時不需要真實邊界框或檢測器，並且僅調整共享投影層，視覺和語言骨幹網絡保持凍結。在K=64的情況下，SaMer移除了超過93%的圖像端令牌，並將ColPali存儲減少16.09倍，同時在Flickr30K和MSCOCO上提升了R@1。這些提升源於物體感知合併保留了查詢可選擇的物體證據，而修剪或僅特徵池化可能移除或壓縮這些證據。SaMer還優於壓縮基準方法，並展現出更強的短語級別定位能力，表明高效的多向量檢索不僅取決於減少令牌數量，還在於保留未來查詢令牌需要選擇的證據。

English

Multi-vector vision-language retrieval preserves fine-grained visual evidence through maximum-similarity late interaction, but dense image-side tokens make storage and scoring expensive. Existing token compression methods reduce this cost, yet they can remove or collapse object- and region-level evidence that future query tokens may need to select. We propose SaMer, an object-aware token merging framework that compresses image-side post-projector tokens into K representative centroids while preserving the original late-interaction interface. SaMer uses object annotations only during training as a merge prior to discourage cross-instance mixing, requires no ground-truth bounding boxes or detectors at inference time, and adapts only the shared projection layer with frozen vision and language backbones. With K=64, SaMer removes more than 93% of image-side tokens and reduces ColPali storage by 16.09times, while improving R@1 on Flickr30K and MSCOCO. These gains arise because object-aware merging preserves query-selectable object evidence that pruning or feature-only pooling can remove or collapse. SaMer also outperforms compression baselines and shows stronger phrase-level grounding, suggesting that efficient multi-vector retrieval depends not only on reducing token count, but on preserving the evidence future query tokens need to select.