所有视觉令牌都同等重要吗？面向视觉-语言检索的保留对象证据的令牌合并

摘要

多向量视觉-语言检索通过最大相似度后期交互保留了细粒度的视觉证据，但密集的图像端令牌导致存储和评分成本高昂。现有的令牌压缩方法降低了这一成本，却可能移除或坍塌未来查询令牌需要选择的物体和区域级证据。我们提出SaMer，一种对象感知的令牌合并框架，将图像端投影后的令牌压缩为K个代表性质心，同时保留原有的后期交互接口。SaMer在训练时仅将物体标注作为合并先验，以抑制跨实例混合，推理时无需真实边界框或检测器，并且仅适配共享投影层，视觉和语言主干网络保持冻结。当K=64时，SaMer移除了超过93%的图像端令牌，并将ColPali的存储量减少了16.09倍，同时在Flickr30K和MSCOCO上提高了R@1指标。这些增益源于对象感知合并保留了查询可选择的物体证据，而剪枝或仅特征池化可能移除或坍塌这些证据。SaMer还优于压缩基线方法，并展现出更强的短语级定位能力，表明高效的多向量检索不仅取决于减少令牌数量，还取决于保留未来查询令牌需要选择的证据。

English

Multi-vector vision-language retrieval preserves fine-grained visual evidence through maximum-similarity late interaction, but dense image-side tokens make storage and scoring expensive. Existing token compression methods reduce this cost, yet they can remove or collapse object- and region-level evidence that future query tokens may need to select. We propose SaMer, an object-aware token merging framework that compresses image-side post-projector tokens into K representative centroids while preserving the original late-interaction interface. SaMer uses object annotations only during training as a merge prior to discourage cross-instance mixing, requires no ground-truth bounding boxes or detectors at inference time, and adapts only the shared projection layer with frozen vision and language backbones. With K=64, SaMer removes more than 93% of image-side tokens and reduces ColPali storage by 16.09times, while improving R@1 on Flickr30K and MSCOCO. These gains arise because object-aware merging preserves query-selectable object evidence that pruning or feature-only pooling can remove or collapse. SaMer also outperforms compression baselines and shows stronger phrase-level grounding, suggesting that efficient multi-vector retrieval depends not only on reducing token count, but on preserving the evidence future query tokens need to select.