PACE：代理能力评估的代理指标

摘要

在SWE-Bench和GAIA等基准上评估LLM代理既昂贵又耗时，还需复杂的基础设施支持。一次评估可能花费数千美元并需数天才能完成。相比之下，测试单一能力（如推理、代码生成）的非代理LLM基准则运行快速且成本低廉。本文探究能否通过一组精心挑选的少量原子化评估实例的性能，准确预测昂贵的代理基准上的表现。为此，我们提出PACE框架，通过从现有非代理评测中选取实例构建代理预测基准——这些实例的聚合分数能最可靠地预测模型在代理基准上的表现。给定一组覆盖原子化能力的候选实例，PACE拟合一个回归模型，将模型在紧凑源实例子集上的得分映射到目标代理基准的得分。该子集通过结合两种互补的实例选择策略（目标相关性局部选择与全局信息性全局选择）来筛选。我们将PACE应用于本文涉及的4个目标代理基准，生成了具体的代理预测基准PACE-Bench。针对14个模型、4个代理基准和19个非代理基准的实验表明，PACE-Bench的代理分数预测留一法交叉验证（LOOCV）平均绝对误差（MAE）低于4%，斯皮尔曼相关系数超过0.80，成对模型排序准确率约达85%，而成本仅为完整代理评估的不到1%。我们进一步分析了所选代理实例，揭示了每个代理基准所独特要求的能力。PACE使从业者能够在模型开发、选择和路由过程中，无需进行完整的代理评估即可获得可靠的代理性能估计。

English

Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure. A single evaluation can cost thousands of dollars and take days to complete. In contrast, non-agentic LLM benchmarks that test individual capabilities (e.g., reasoning, code generation) are fast and cheap to run. In this paper, we investigate whether performance on expensive agentic benchmarks can be accurately predicted by the performance on a small, carefully selected subset of atomic evaluation instances. We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations whose aggregate scores most reliably predict model performances on agentic benchmarks. Given a pool of candidate instances spanning atomic capabilities, PACE fits a regression that maps a model's scores on a compact subset of source instances to its score on the target agentic benchmark. The subset itself is curated by combining two complementary instance-selection strategies, target-relevance local selection and globally informative global selection. We apply PACE to the 4 target agentic benchmarks in this paper, which yields PACE-Bench, the concrete proxy benchmark that we evaluate in the paper. Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost. We further analyze the selected proxy instances, revealing which skills each agentic benchmark uniquely demands. PACE enables practitioners to obtain reliable estimates of agentic performance during model development, selection, and routing, without the overhead of full agent evaluation.