長視界終端基準測試：以密集獎勵評分挑戰智能體在長期任務中的極限

摘要

AI代理已能自主完成短時間內定義明確的任務。然而，現有的終端基準評測大多聚焦於幾分鐘內可完成的簡單問題，且僅以最終結果進行評估。這種設定忽略了中間進度與部分解決方案，導致獎勵訊號稀疏，無法完整呈現代理能力。我們提出「Long-Horizon-Terminal-Bench」，這是一個包含46項長週期任務的終端基準評測，涵蓋九大類別，包括實驗複現、軟體工程、多模態分析、互動遊戲與科學計算。每項任務均採用Terminal-Bench風格的設定，附帶參考解決方案或模擬引擎，但進一步拆解為細粒度的分級子任務。此設計能提供密集的中間獎勵與部分分數，使評估不僅能判斷代理是否達成最終目標，還能反映其在開放式工作流程中的進展程度。「Long-Horizon-Terminal-Bench」中的任務通常需要數百回合執行，耗時數分鐘至數小時，著重測試長週期規劃、長上下文管理與反覆除錯能力，而非一次性問題解決。我們評估了15個前沿模型，發現代理平均每項任務消耗990萬個token，約執行231回合及85.3分鐘，使「Long-Horizon-Terminal-Bench」比先前基於終端的基準評測更具挑戰性。即使是最強的受測模型，在部分獎勵閾值0.95下僅達到15.2%的pass@1，在完美獎勵閾值1.0下為10.9%；而所有模型在兩種閾值下的平均通過率分別為4.3%與1.7%。這些結果顯示仍有極大改進空間。我們進一步分析失敗模式與錯誤類型，並釋出「Long-Horizon-Terminal-Bench」，以支援未來在長週期終端代理領域的進展。

English

AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.

長視界終端基準測試：以密集獎勵評分挑戰智能體在長期任務中的極限

Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

摘要

Support