EarlyEval：基于早期结果预测的低成本智能体评估

摘要

评估LLM智能体对于指导其发展至关重要，然而这一过程已变得成本高昂得令人望而却步：前沿模型在智能体基准测试上单次运行就可能耗费数百至数千美元，且在迭代开发周期中这一代价会被反复支付。以往的研究工作以基准蒸馏为核心，减少了评估任务的数量，但并未降低执行每个保留任务的成本。在本工作中，我们引入早期结果预测（early outcome prediction），这是一条互补的效率优化路径，转而削减每个任务内部的成本。我们的关键洞察在于：智能体的最终结果往往在执行完成之前就能从其中间行为中明显看出。我们将该思想实例化为EarlyEval——一个轻量级框架，它训练一对基于行为、文本和参考答案特征的LightGBM成功与失败分类器，并在任一分类器越过校准后的置信度阈值时立即终止智能体运行，每步仅增加可忽略不计的开销。在SWE-bench Verified、TerminalBench和Toolathlon三个基准测试上，EarlyEval能够以89%-97%的预测准确率消除13%-26%的智能体执行步骤，并节省高达44.1%的输入token和29.4%的输出token，同时仅使各智能体的平均解决率产生一至两个百分点的扰动。

English

Evaluating LLM agents is essential for guiding their development, yet it has grown prohibitively expensive: a single pass of a frontier model over an agentic benchmark can cost hundreds to thousands of dollars, a price paid repeatedly across iterative development cycles. Prior efforts, centered on benchmark distillation, reduce the number of evaluation tasks but leave the cost of executing each retained task untouched. In this work, we introduce early outcome prediction, a complementary axis of efficiency that instead cuts cost within each task. Our key insight is that an agent's final outcome is often evident from its intermediate behavior well before execution completes. We instantiate this idea in EarlyEval, a lightweight framework that trains a pair of LightGBM success and failure classifiers over behavioral, textual, and reference-solution features, and halts an agent run the moment either classifier crosses a calibrated confidence threshold, adding negligible per-step overhead. Across three benchmarks, SWE-bench Verified, TerminalBench, and Toolathlon, EarlyEval can eliminate 13%-26% of agent steps and up to 44.1% input tokens and 29.4% output tokens at 89%-97% prediction accuracy, while perturbing per-agent resolve rates by only one to two percentage points on average.

EarlyEval：基于早期结果预测的低成本智能体评估

EarlyEval: Cheaper Agent Evaluation via Early Outcome Prediction

摘要

Support