LoopArena：模型作为循环工程运行时控制器的基准测试

摘要

循环工程（Loop Engineering）正逐步成为围绕编码智能体组织开发工作的一种实践。与逐条手工编写提示词不同，实践者设计能够监控进度、分配任务、运行检查并决定智能体下一步动作的循环。即使拥有能力强大的编码智能体，循环仍可能信任过时的进度记录、跳过必要的验证、将预算消耗在错误方向上，或在任务尚不宜提交时提前终止。然而，单次端到端运行的最终结果无法区分成败究竟源自循环的引导，还是编码智能体执行任务的能力。我们提出 LoopArena，一个用于评估模型在长期任务中引导独立编码智能体的能力的基准。受评估的模型称为控制器（Controller）：在每轮编码之后，它接收运行的结构化摘要，并指示一个独立的、固定的编码智能体——工作器（Worker）——下一步应执行或验证什么，或者决定是否终止。LoopArena 在三种在执行范围与成本上各有差异的互补设定下评估这一能力。第一类（Type I）通过经执行验证的问题对下一步循环合约（Loop Contract）的选择进行评分，评估时无需运行工作器。第二类（Type II）对完整任务的一个选定切片执行重复控制，而第三类（Type III）则从任务的原始状态出发评估配对的完整任务。在完整任务上，观测到的最高严格成功率为 24.69%，表明长时程循环控制仍有很大的改进空间。在所有控制器中，估计推理成本的成对降低平均为 64.4%，且第二类在核心（Core）主要指标下产生了相似的排序（斯皮尔曼相关系数 \(ρ=0.9747\)）。我们在 https://github.com/AMAP-ML/LoopArena 发布了基准数据与评估代码。

English

Loop Engineering is emerging as a practice for organizing development work around coding agents. Instead of writing each prompt by hand, practitioners design loops that monitor progress, assign work, run checks, and decide what the agent should do next. Even with a capable coding agent, a loop may trust a stale progress note, skip needed verification, spend its budget in the wrong direction, or stop before the task is safe to submit. Yet the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task. We introduce LoopArena, a benchmark for evaluating how well one model can guide a separate coding agent through a long-running task. The model under evaluation is the Controller: after each coding round, it receives a structured summary of the run and instructs a separate, fixed coding agent, the Worker, on what to do or verify next, or decides whether to stop. LoopArena evaluates this ability in three complementary settings that differ in execution scope and cost. Type I scores next-step Loop Contract selection through execution-validated questions without running the Worker at evaluation time. Type II executes repeated control over a selected slice of a full task, while Type III evaluates the paired full task from its original state. On full tasks, the best observed Strict Success Rate is 24.69\%, leaving substantial room for improvement in long-horizon loop control. Across Controllers, the paired reduction in estimated inference cost averages 64.4\%, and Type II produces a similar ordering under the main Core criterion (Spearman's \(ρ=0.9747\)). We release the benchmark data and evaluation code at https://github.com/AMAP-ML/LoopArena .

LoopArena：模型作为循环工程运行时控制器的基准测试

LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering

摘要

Support