VisR-Bench：多語言長文件理解中視覺檢索增強生成之實證研究

摘要

世界上大多數組織數據都以文檔形式存儲，而視覺檢索在從這些文檔中釋放集體智慧方面起著至關重要的作用。然而，現有的基準測試主要集中在僅限於英文的文檔檢索，或僅考慮單頁圖像上的多語言問答。為彌補這一差距，我們引入了VisR-Bench，這是一個專為長文檔中問題驅動的多模態檢索設計的多語言基準測試。我們的基準測試包含超過35,000個高質量問答對，涵蓋1,200份文檔，能夠對多模態檢索進行細粒度評估。VisR-Bench涵蓋十六種語言，包含三種問題類型（圖表、文本和表格），提供了多樣化的語言和問題覆蓋範圍。與之前的數據集不同，我們引入了沒有明確答案的查詢，防止模型依賴於表面的關鍵詞匹配。我們評估了各種檢索模型，包括基於文本的方法、多模態編碼器和多模態大語言模型（MLLMs），深入探討了它們的優勢和局限性。我們的結果表明，儘管MLLMs顯著優於基於文本和多模態編碼器的模型，但在處理結構化表格和低資源語言時仍存在困難，這凸顯了多語言視覺檢索中的關鍵挑戰。

English

Most organizational data in this world are stored as documents, and visual retrieval plays a crucial role in unlocking the collective intelligence from all these documents. However, existing benchmarks focus on English-only document retrieval or only consider multilingual question-answering on a single-page image. To bridge this gap, we introduce VisR-Bench, a multilingual benchmark designed for question-driven multimodal retrieval in long documents. Our benchmark comprises over 35K high-quality QA pairs across 1.2K documents, enabling fine-grained evaluation of multimodal retrieval. VisR-Bench spans sixteen languages with three question types (figures, text, and tables), offering diverse linguistic and question coverage. Unlike prior datasets, we include queries without explicit answers, preventing models from relying on superficial keyword matching. We evaluate various retrieval models, including text-based methods, multimodal encoders, and MLLMs, providing insights into their strengths and limitations. Our results show that while MLLMs significantly outperform text-based and multimodal encoder models, they still struggle with structured tables and low-resource languages, highlighting key challenges in multilingual visual retrieval.

VisR-Bench：多語言長文件理解中視覺檢索增強生成之實證研究

VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding

摘要

Support