ConceptFormer：视觉文档检索中面向查询-文档对齐的自适应潜在概念学习

摘要

视觉文档检索是多模态检索增强生成的一个关键组成部分，旨在从文档集合中识别与查询相关的页面，其中证据分布在文本、布局、图表和视觉结构中。近期为获得更细粒度监督的尝试主要依赖文本描述或局部视觉区域作为证据代理。然而，此类监督信号可能忽略复杂的视觉结构，或对底层证据产生不完整和不准确的表示。为解决这些局限性，我们提出了ConceptFormer，一种用于视觉文档检索的潜在概念表示学习框架。ConceptFormer将查询相关证据建模为连续的、以查询为条件的潜在概念，明确桥接局部视觉证据与语义相关性，既不需要文本中间表示，也不直接依赖原始视觉标注。在训练过程中，ConceptFormer使用强大的视觉语言模型动态确定潜在概念标记的数量，并将这些概念作为中间表示来弥合查询与文档之间的语义鸿沟，从而引导嵌入空间的学习。在多个视觉文档检索基准上的实验表明，ConceptFormer相比最强的视觉检索基线和最强的基于OCR的文本检索基线，平均NDCG@10分别取得了16.7%和22.1%的相对提升。进一步分析表明，潜在概念能够有效连接局部视觉证据与语义相关性，使检索器既能捕获细粒度文本线索，又能理解复杂的文档级视觉结构，同时保持强大的检索对齐能力。代码和数据可在https://github.com/Neuir/ConceptFormer获取。

English

Visual document retrieval is a critical component of multimodal retrieval-augmented generation, aiming to identify query-relevant pages from document collections where evidence is distributed across text, layout, charts, and visual structures. Recent efforts toward finer-grained supervision primarily rely on textual descriptions or localized visual regions as evidence proxies. However, such supervision signals may either overlook complex visual structures or provide incomplete and inaccurate representations of the underlying evidence. To address these limitations, we propose ConceptFormer, a latent concept representation learning framework for visual document retrieval. ConceptFormer models query-relevant evidence as continuous, query-conditioned latent concepts that explicitly bridge localized visual evidence and semantic relevance, without requiring either textual intermediate representations or direct reliance on raw visual annotations. During training, ConceptFormer employs a strong vision-language model to dynamically determine the number of latent concept tokens and uses these concepts as an intermediate representation to bridge the semantic gap between queries and documents, thereby guiding the learning of the embedding space. Experiments on diverse visual document retrieval benchmarks demonstrate that ConceptFormer achieves 16.7\% and 22.1\% relative improvements in average NDCG@10 over the strongest visual retrieval baseline and the strongest OCR-based text retrieval baseline, respectively. Further analysis reveals that latent concepts effectively connect localized visual evidence with semantic relevance, enabling the retriever to capture both fine-grained textual cues and complex document-level visual structures while preserving strong retrieval alignment. Codes and data are available at https://github.com/Neuir/ConceptFormer.