ExtractBench：一個用於模式引導企業文件抽取的基準

摘要

企業工作流程日益依賴代理程式進行結構化綱要引導的抽取：給定一份文件與使用者定義的結構化綱要，代理程式忠實遵循綱要，以來源證據作為溯源中繼資料，產出正確的輸出。我們提出 ExtractBench，一個用於結構化綱要引導抽取的基準，且據我們所知，是首個同時評量值準確度、大規模記錄完整性、溯源與實測成本的基準。評量系統包含 370 份企業文件共 4,869 頁，涵蓋 8 個商業領域與 67 種文件類型，並以明確標籤區分不同挑戰情境。可擴展的結構化綱要與真實答案建置流程結合了真實文件的獨立系統一致性、合成清單的已知值，以及表單的人工驗證。我們報告用於值準確度的順序不敏感值 F1，以及兩個用於來源可追溯性的溯源指標：詞級與頁級 F1。商業 VLM 在短文件上表現良好，但在長文件上經常截斷記錄清單；而編碼代理程式則能以高得多的成本維持較高準確度。LlamaExtract Agentic Plus 在所有三項指標上排名第一，其準確度與編碼代理程式相當，但成本僅為其一小部分。資料集與評量程式碼可在 https://huggingface.co/datasets/llamaindex/ExtractBench{HuggingFace} 與 https://github.com/run-llama/ExtractBench{GitHub} 取得。

English

Enterprise workflows increasingly rely on agents for schema-guided extraction: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on https://huggingface.co/datasets/llamaindex/ExtractBench{HuggingFace} and https://github.com/run-llama/ExtractBench{GitHub}.