超越视觉相似性：面向基于知识的视觉问答的实体对齐检索

摘要

基于知识的视觉问答（KB-VQA）依赖于检索外部信息来回答涉及长尾实体的查询。然而，现有的检索流程主要采用CLIP风格的双编码器，其优先考虑表面视觉相似性而非实体级语义对齐。当语义相同的概念表现出较大的视觉差异，或不同实体在视觉上相似时，这种范式往往失效。为解决这一问题，我们提出KBMR——首个专为KB-VQA设计的基于MLLM的嵌入检索器。KBMR利用多模态大语言模型（MLLM）强大的自回归能力，将图像映射到能更好保持概念身份的语义空间中。为应对维基百科规模检索中噪声监督的挑战，我们引入了一种基于MLLM的语义判别器，用于生成连续的实体一致性权重。这些权重引导了一种新颖的连续语义蒸馏目标，从而在刚性二元标签之外实现有效的难负样本采样和软监督。大量实验表明，KBMR显著优于CLIP基线，在检索Recall@1上最高提升14.7%，在端到端VQA准确率上提升9.4%。代码已在 https://github.com/realHarryX/KBMR 公开。

English

Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieving external information to answer queries involving long-tail entities. However, existing retrieval pipelines predominantly employ CLIP-style dual encoders, which prioritize surface-level visual similarity over entity-level semantic alignment. This paradigm often fails when semantically identical concepts exhibit large visual variations or when distinct entities appear visually similar. To address this, we propose KBMR, the first MLLM-based embedding retriever tailored for KB-VQA. Leveraging the robust autoregressive capabilities of MLLMs, KBMR maps images into a semantic space that better preserves concept identity. To tackle the challenge of noisy supervision in Wikipedia-scale retrieval, we introduce an MLLM-based semantic discriminator that generates continuous entity-consistency weights. These weights guide a novel continuous semantic distillation objective, enabling effective hard negative sampling and soft supervision beyond rigid binary labels. Extensive experiments demonstrate that KBMR significantly outperforms CLIP baselines, yielding up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. Code is available at https://github.com/realHarryX/KBMR.