EarlyEval：透過早期結果預測實現更經濟的代理評測

摘要

評估 LLM 代理對於引導其開發至關重要，然而其成本已成長到令人卻步的程度：前沿模型在代理型基準上一次單獨運行就可能耗費數百至數千美元，而這筆代價會在迭代開發週期中被反覆支付。先前以基準蒸餾為核心的方法，雖能減少評估任務的數量，卻未觸及執行每個保留任務本身的成本。在本研究中，我們提出早期結果預測（early outcome prediction），這是一個互補的效率軸向，改從每個任務內部削減成本。我們的關鍵洞察在於：代理的最終結果往往在執行完成之前，就能從其中間行為中清楚浮現。我們將此概念落實於 EarlyEval 框架——這是一個輕量級框架，訓練一對分別針對成功與失敗的 LightGBM 分類器，特徵涵蓋行為、文本及參考解答資訊；當任一分類器跨越校準後的信心門檻時，便立即中止代理運行，而每步僅增添可忽略的額外開銷。在 SWE-bench Verified、TerminalBench 與 Toolathlon 三個基準上，EarlyEval 能以 89%–97% 的預測準確率消除 13%–26% 的代理步驟，並最高節省 44.1% 的輸入 token 與 29.4% 的輸出 token，同時平均僅使每個代理的解決率變動一至兩個百分點。

English

Evaluating LLM agents is essential for guiding their development, yet it has grown prohibitively expensive: a single pass of a frontier model over an agentic benchmark can cost hundreds to thousands of dollars, a price paid repeatedly across iterative development cycles. Prior efforts, centered on benchmark distillation, reduce the number of evaluation tasks but leave the cost of executing each retained task untouched. In this work, we introduce early outcome prediction, a complementary axis of efficiency that instead cuts cost within each task. Our key insight is that an agent's final outcome is often evident from its intermediate behavior well before execution completes. We instantiate this idea in EarlyEval, a lightweight framework that trains a pair of LightGBM success and failure classifiers over behavioral, textual, and reference-solution features, and halts an agent run the moment either classifier crosses a calibrated confidence threshold, adding negligible per-step overhead. Across three benchmarks, SWE-bench Verified, TerminalBench, and Toolathlon, EarlyEval can eliminate 13%-26% of agent steps and up to 44.1% input tokens and 29.4% output tokens at 89%-97% prediction accuracy, while perturbing per-agent resolve rates by only one to two percentage points on average.