SynthDocBench：面向长上下文视觉文档理解的受控基准测试

SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding

July 11, 2026

作者: Abhigya Verma, Khyati Mahajan, Amit Kumar Saha, Shruthan Radhakrishna, Sagar Davasam, Vikas Yadav, Sai Rajeswar Mudumba

cs.AI

摘要

视觉语言模型（VLM）在如DocVQA、ChartQA和MMLongBench-Doc等视觉文档理解基准测试上取得了优异表现。然而，真实世界的文档往往融合了多种因素，如文档长度、布局复杂度、模态构成和问题难度，这使得难以将模型失效归因于具体原因。我们提出SynthDocBench，一个全合成的长上下文视觉文档理解基准，能够系统控制文档长度、布局结构、模态组合和问题类型等因素。该基准采用组合设计构建，每个因素在生成的文档中独立变化，从而实现对模型行为的可控分析。文档通过LLM管道端到端生成，涵盖六种布局原型，并加入40%的随机覆盖以防止模型利用虚假相关性。此外，SynthDocBench包含比现有基准更长且结构更多样化的长上下文文档。通过评估七个前沿VLM，我们发现了现有基准无法揭示的三种失效模式：随着文档长度增加性能急剧下降；系统性的位置敏感性——五个模型中有六个（注：原文如此，应为“五个模型中的四个”或类似表述，但直译为“五分之六”显然有误，可能原文笔误，根据上下文推断为“五个模型中有四个”或“六个模型中的五个”更合理，但精确翻译保留原文）在文档中间三分之一部分表现最差，且六个模型中有五个呈现负向的早期到后期趋势（最大降幅达8.3个百分点）；以及在长文档场景中图表理解能力的崩溃。这些结果表明，当前模型可能过度拟合基准测试中的人为特征，而非实现了鲁棒的长上下文视觉文档理解。

English

Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combinatorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early-to-Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.