LoopArena：將模型作為迴圈工程的運行時控制器進行基準測試

摘要

迴圈工程（Loop Engineering）正逐漸成為一種圍繞編碼代理組織開發工作的實務做法。與其逐一手工編寫提示詞，從業者設計迴圈來監控進度、分配工作、執行檢查，並決定代理下一步該做什麼。即使擁有能力強大的編碼代理，迴圈仍可能信任過時的進度記錄、跳過必要的驗證、將預算浪費在錯誤的方向上，或在任務尚未達到可安全提交的狀態前就提前停止。然而，單次端到端執行的最終結果無法分辨成功或失敗究竟反映了迴圈的引導能力，還是編碼代理執行任務的能力。我們提出 LoopArena，一個用於評估單一模型能否妥善引導另一個獨立編碼代理完成長時間執行任務的基準。受評估的模型稱為控制器（Controller）：在每一輪編碼結束後，它接收執行的結構化摘要，指示另一個固定的獨立編碼代理——工作代理（Worker）——接下來應執行或驗證什麼，或決定是否停止。LoopArena 在三個執行範圍與成本各異的互補設定中評估此能力。Type I 透過經執行驗證的問題，對下一步迴圈合約（Loop Contract）的選擇進行評分，評估時無需實際執行工作代理。Type II 對完整任務的選定片段執行重複控制，而 Type III 則從任務的原始狀態評估配對的完整任務。在完整任務上，觀察到的最佳嚴格成功率（Strict Success Rate）為 24.69%，顯示長時間跨度迴圈控制仍有大幅改進空間。在各個控制器之間，估計推論成本的配對平均降低幅度為 64.4%，且 Type II 在主要核心指標下產生了相似的排序（斯皮爾曼 ρ=0.9747）。我們在 https://github.com/AMAP-ML/LoopArena 發布基準資料與評估程式碼。

English

Loop Engineering is emerging as a practice for organizing development work around coding agents. Instead of writing each prompt by hand, practitioners design loops that monitor progress, assign work, run checks, and decide what the agent should do next. Even with a capable coding agent, a loop may trust a stale progress note, skip needed verification, spend its budget in the wrong direction, or stop before the task is safe to submit. Yet the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task. We introduce LoopArena, a benchmark for evaluating how well one model can guide a separate coding agent through a long-running task. The model under evaluation is the Controller: after each coding round, it receives a structured summary of the run and instructs a separate, fixed coding agent, the Worker, on what to do or verify next, or decides whether to stop. LoopArena evaluates this ability in three complementary settings that differ in execution scope and cost. Type I scores next-step Loop Contract selection through execution-validated questions without running the Worker at evaluation time. Type II executes repeated control over a selected slice of a full task, while Type III evaluates the paired full task from its original state. On full tasks, the best observed Strict Success Rate is 24.69\%, leaving substantial room for improvement in long-horizon loop control. Across Controllers, the paired reduction in estimated inference cost averages 64.4\%, and Type II produces a similar ordering under the main Core criterion (Spearman's \(ρ=0.9747\)). We release the benchmark data and evaluation code at https://github.com/AMAP-ML/LoopArena .

LoopArena：將模型作為迴圈工程的運行時控制器進行基準測試

LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering

摘要

Support