ExtractBench：面向架构引导的企业文档抽取基准

摘要

企业工作流日益依赖智能体进行模式引导的抽取：给定一个文档和用户定义的模式，智能体忠实地遵循该模式，生成正确的输出，并以源证据作为溯源元数据。我们提出了 ExtractBench，一个用于模式引导抽取的基准；据我们所知，这是首个同时评估值准确性、大规模记录完整性、溯源性和实测成本的基准。该评估数据集包含 4,869 页，涵盖 370 份企业文档、8 个业务领域和 67 种文档类型，并带有清晰标签以区分各自面临的挑战场景。可扩展的模式与真值整理流程融合了真实文档的独立系统一致性、合成列表的已知值以及表单的人工验证。我们采用顺序不敏感的值 F1 来评估值准确性，并采用两个溯源指标（词级 F1 和页面级 F1）来评估源可追溯性。商用视觉语言模型在短文档上表现良好，但在长文档上经常截断记录列表；而编程智能体则以高得多的成本保持了更高的准确性。LlamaExtract Agentic Plus 在所有三项指标上均排名第一，其准确性与编程智能体相当，而成本仅为后者的一小部分。数据集和评估代码可在 https://huggingface.co/datasets/llamaindex/ExtractBench{HuggingFace} 和 https://github.com/run-llama/ExtractBench{GitHub} 获取。

English

Enterprise workflows increasingly rely on agents for schema-guided extraction: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on https://huggingface.co/datasets/llamaindex/ExtractBench{HuggingFace} and https://github.com/run-llama/ExtractBench{GitHub}.