超越視覺相似性：面向知識型視覺問答的實體對齊檢索

摘要

基於知識的視覺問答（KB-VQA）依賴檢索外部資訊來回答涉及長尾實體的查詢。然而，現有的檢索流程大多採用 CLIP 風格的雙編碼器，這類方法優先考慮表面層級的視覺相似性，而非實體層級的語義對齊。當語義相同的概念呈現出巨大的視覺差異，或不同實體在視覺上相似時，此範式往往會失效。為了解決這個問題，我們提出 KBMR——第一個專為 KB-VQA 設計、基於 MLLM 的嵌入檢索器。KBMR 利用 MLLM 強大的自迴歸能力，將影像映射到更能保留概念身分的語義空間。為了應對維基百科規模檢索中雜訊監督的挑戰，我們引入了基於 MLLM 的語義判別器，用以產生連續的實體一致性權重。這些權重引導一種新穎的連續語義蒸餾目標，使得在僵化的二元標籤之外，也能實現有效的困難負樣本採樣與軟監督。大量實驗表明，KBMR 顯著優於 CLIP 基線方法，在檢索 Recall@1 上最高提升 14.7%，在端到端 VQA 準確率上則有 9.4% 的提升。程式碼可於 https://github.com/realHarryX/KBMR 取得。

English

Knowledge-Based Visual Question Answering (KB-VQA) relies on retrieving external information to answer queries involving long-tail entities. However, existing retrieval pipelines predominantly employ CLIP-style dual encoders, which prioritize surface-level visual similarity over entity-level semantic alignment. This paradigm often fails when semantically identical concepts exhibit large visual variations or when distinct entities appear visually similar. To address this, we propose KBMR, the first MLLM-based embedding retriever tailored for KB-VQA. Leveraging the robust autoregressive capabilities of MLLMs, KBMR maps images into a semantic space that better preserves concept identity. To tackle the challenge of noisy supervision in Wikipedia-scale retrieval, we introduce an MLLM-based semantic discriminator that generates continuous entity-consistency weights. These weights guide a novel continuous semantic distillation objective, enabling effective hard negative sampling and soft supervision beyond rigid binary labels. Extensive experiments demonstrate that KBMR significantly outperforms CLIP baselines, yielding up to a 14.7% improvement in retrieval Recall@1 and a 9.4% gain in end-to-end VQA accuracy. Code is available at https://github.com/realHarryX/KBMR.