长视界终端任务基准：采用密集奖励评分机制测试智能体在长视界终端任务中的极限

摘要

AI智能体已能自主完成简短且定义明确的任务。然而，现有的终端基准主要聚焦于几分钟内可解决的简单问题，且仅通过最终结果进行评估。这种设置忽略了中间进展和部分解决方案，导致奖励信号稀疏，无法全面反映智能体能力。为此，我们提出长时终端基准（Long-Horizon-Terminal-Bench），这是一个包含46项长时任务的终端基准，覆盖实验复现、软件工程、多模态分析、交互游戏和科学计算等九大类别。每项任务均遵循终端基准风格，配有参考解决方案或仿真引擎，但进一步分解为细粒度的分级子任务。这一设计支持密集的中间奖励和部分得分，使评估不仅能判断智能体是否达成最终目标，还能衡量其在开放式工作流中的进展程度。该基准中的任务通常需要数百个回合和数分钟至数小时的执行时间，强调长时规划、长上下文管理及迭代调试，而非一次性的问题解决。我们对15个前沿模型进行了评估，发现智能体每项任务平均消耗990万token，每轮运行约需231个回合和85.3分钟的执行时间，这使得长时终端基准比此前基于终端的基准更具挑战性。即使是最强的测试模型，在部分奖励阈值为0.95时，pass@1也仅为15.2%，在完美奖励阈值1.0下则为10.9%；而所有模型在这两个阈值下的平均通过率分别为4.3%和1.7%。这些结果表明仍有巨大改进空间。我们进一步分析了失败模式和错误类型，并开源长时终端基准，以支持长时终端智能体的未来发展。

English

AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.

长视界终端任务基准：采用密集奖励评分机制测试智能体在长视界终端任务中的极限

Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

摘要

Support