ConceptFormer：面向視覺文件檢索中查詢-文件對齊的自適應潛在概念學習

摘要

視覺文件檢索是多模態檢索增強生成的關鍵組成部分，旨在從文件集合中識別與查詢相關的頁面，而這些文件中的證據分佈於文字、版面、圖表及視覺結構之中。近期追求更細粒度監督的研究，主要依賴文字描述或局部視覺區域作為證據代理。然而，此類監督訊號可能忽略複雜的視覺結構，或對潛在證據提供不完整且不精確的表示。為了解決這些限制，我們提出了ConceptFormer，一種應用於視覺文件檢索的潛在概念表示學習框架。ConceptFormer將查詢相關的證據建模為連續的、查詢條件化的潛在概念，這些概念明確地橋接了局部視覺證據與語義相關性，既不需要文字中介表示，也無須直接依賴原始視覺註釋。在訓練期間，ConceptFormer利用強大的視覺語言模型動態決定潛在概念標記的數量，並將這些概念作為中間表示，以縮小查詢與文件之間的語義差距，從而引導嵌入空間的學習。在多樣化的視覺文件檢索基準上的實驗表明，相較於最強的視覺檢索基線與最強的基於OCR的文字檢索基線，ConceptFormer在平均NDCG@10上分別實現了16.7\%和22.1\%的相對提升。進一步的分析揭示，潛在概念能有效連接局部視覺證據與語義相關性，使檢索器在保持強大檢索對齊的同時，能捕捉細粒度的文字線索與複雜的文件級視覺結構。程式碼與資料可在 https://github.com/Neuir/ConceptFormer 取得。

English

Visual document retrieval is a critical component of multimodal retrieval-augmented generation, aiming to identify query-relevant pages from document collections where evidence is distributed across text, layout, charts, and visual structures. Recent efforts toward finer-grained supervision primarily rely on textual descriptions or localized visual regions as evidence proxies. However, such supervision signals may either overlook complex visual structures or provide incomplete and inaccurate representations of the underlying evidence. To address these limitations, we propose ConceptFormer, a latent concept representation learning framework for visual document retrieval. ConceptFormer models query-relevant evidence as continuous, query-conditioned latent concepts that explicitly bridge localized visual evidence and semantic relevance, without requiring either textual intermediate representations or direct reliance on raw visual annotations. During training, ConceptFormer employs a strong vision-language model to dynamically determine the number of latent concept tokens and uses these concepts as an intermediate representation to bridge the semantic gap between queries and documents, thereby guiding the learning of the embedding space. Experiments on diverse visual document retrieval benchmarks demonstrate that ConceptFormer achieves 16.7\% and 22.1\% relative improvements in average NDCG@10 over the strongest visual retrieval baseline and the strongest OCR-based text retrieval baseline, respectively. Further analysis reveals that latent concepts effectively connect localized visual evidence with semantic relevance, enabling the retriever to capture both fine-grained textual cues and complex document-level visual structures while preserving strong retrieval alignment. Codes and data are available at https://github.com/Neuir/ConceptFormer.