AGORA：基於檔案的智能體式職場文件推理基準

摘要

大型语言模型正越来越多地被部署为智能体，通过推理文档而非依赖参数知识来回答问题。我们研究的是基于档案的推理：在庞大而杂乱的工作文件集合中定位稀疏证据，协调不一致的术语、单位和时间惯例，并计算出答案。现有基准只覆盖了该场景的部分方面，没有一项能同时强调档案基础性、智能体探索以及跨领域覆盖。我们提出Agora基准，该基准将362个问题与八个领域的9,664份真实文档（共计3.72亿token）配对，规模远超任何模型的上下文窗口，因此智能体必须审慎探索而非穷尽扫描。Agora通过一条智能体流水线构建，该流水线结合了跨文档任务合成、防泄露混淆与难度过滤。在评估八个模型后，我们发现该任务远未解决：即使最强的模型也仅达到59.4%的准确率，且各领域间差异显著。

English

Large language models are increasingly deployed as agents that reason over documents rather than answer from parametric knowledge. We study archive-grounded reasoning: locating sparse evidence across a large, messy collection of workplace files, reconciling inconsistent terminology, units, and time conventions, and computing an answer. Existing benchmarks address only parts of this setting and none jointly stresses archive-groundedness, agentic exploration, and cross-domain coverage. We introduce Agora, a benchmark pairing 362 questions with eight domain collections of 9,664 authentic documents and 372M tokens, far exceeding any model's context window, so agents must explore deliberately rather than scan exhaustively. Agora is built by an agentic pipeline combining cross-document task synthesis, leakage-preventing obfuscation, and difficulty filtering. Evaluating eight models, we find the task far from solved: even the strongest reaches only 59.4% accuracy, with notable variation across domains.