PACE：代理能力評估的代理指標

摘要

評估LLM代理（Agent）於SWE-Bench與GAIA等基準測試時，可能耗費高昂成本、大量時間，且需複雜基礎設施。單次評估可能花費數千美元，並需數日才能完成。相較之下，測試單一能力（如推理、程式碼生成）的非代理型LLM基準測試則快速且廉價。本文探討是否能透過少量且經謹慎挑選的原子評估實例（instance）之表現，準確預測成本高昂的代理型基準測試結果。我們提出PACE框架，透過從現有非代理型評估中選取實例，建構代理型基準測試的代理基準（proxy benchmark），其總分最能可靠預測模型於代理型基準測試的表現。給定一組涵蓋原子能力的候選實例池，PACE擬合回歸模型，將模型在一小部分來源實例上的分數映射至目標代理型基準測試的分數。該子集本身透過結合兩種互補的實例選擇策略（目標相關局部選擇與全域資訊性全域選擇）來篩選。我們將PACE應用於本文中的四個目標代理型基準測試，產生了PACE-Bench，即本文中評估的具體代理基準。針對14個模型、4個代理型基準測試及19個非代理型基準測試的實驗顯示，PACE-Bench在留一法交叉驗證（LOOCV）下，平均絕對誤差（MAE）低於4%、斯皮爾曼相關係數高於0.80、成對模型排名準確率約達85%，而成本僅為完整代理型評估的不到1%。我們進一步分析所選的代理實例，揭示每個代理型基準測試獨特需求的能力。PACE使實務工作者能在模型開發、選擇及路由過程中，無需負擔完整代理評估的開銷，即可獲得可靠的代理表現估計值。

English

Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure. A single evaluation can cost thousands of dollars and take days to complete. In contrast, non-agentic LLM benchmarks that test individual capabilities (e.g., reasoning, code generation) are fast and cheap to run. In this paper, we investigate whether performance on expensive agentic benchmarks can be accurately predicted by the performance on a small, carefully selected subset of atomic evaluation instances. We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations whose aggregate scores most reliably predict model performances on agentic benchmarks. Given a pool of candidate instances spanning atomic capabilities, PACE fits a regression that maps a model's scores on a compact subset of source instances to its score on the target agentic benchmark. The subset itself is curated by combining two complementary instance-selection strategies, target-relevance local selection and globally informative global selection. We apply PACE to the 4 target agentic benchmarks in this paper, which yields PACE-Bench, the concrete proxy benchmark that we evaluate in the paper. Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost. We further analyze the selected proxy instances, revealing which skills each agentic benchmark uniquely demands. PACE enables practitioners to obtain reliable estimates of agentic performance during model development, selection, and routing, without the overhead of full agent evaluation.