執行或不執行：基於LLM的程式修復中程式碼執行的成本效益分析

摘要

基於大型語言模型的程式修復代理，日益建構在「生成-執行-修正」的範式之上，透過反覆執行測試來評估並優化修補程式。此種基於執行的途徑已成為當前最先進系統的標準做法。然而，執行過程既耗時又昂貴，但其對這些代理的影響仍未被充分探討。本文針對LLM導向程式修復中的執行行為，進行了一項兩階段的實證研究。首先，為了大規模描述執行行為的特徵，我們分析了來自SWE-bench排行榜提交資料中的7,745條代理軌跡。其次，我們針對200個SWE-bench實例與三種代理（Claude Code、Codex及開源版OpenCode），在四種執行範式下評估了3,000次端到端的修復嘗試，從而得以進行效能與成本的細緻比較。我們的分析揭示了三個關鍵發現：（1）程式執行在所有分析的代理與模型中均有使用，平均每個任務有8.8次測試運行。不同代理與模型之間的執行行為差異顯著，頻率範圍從每個任務2次到19次不等，且後期階段的執行成功率始終高於早期階段。（2）執行限制對修復成功率的影響甚微：在搭載SOTA模型的商業代理中，「禁止執行」與「無限制」之間的解析率差距僅為1.25個百分點，且不具統計顯著性，而「禁止執行」能大幅節省令牌與實際耗時成本。（3）執行效益具有集中性而非均勻分布。這些模式表明，當前代理對執行採取不加區別的運用，在效益甚微的實例上仍付出其成本。因此，執行應被視為一種具有明確成本效益權衡的資源，而非一項預設能力。

English

LLM-based agents for program repair are increasingly built on a "generate-run-revise" paradigm, iteratively executing tests to evaluate and refine patches. This execution-based approach has become standard practice in state-of-the-art systems. However, executions can be time-consuming and expensive, yet their impact on these agents remains underexplored. In this paper, we conduct a two-stage empirical study over execution behavior in LLM-based program repair. To characterize execution behavior at scale, we first analyze 7,745 agent traces from SWE-bench leaderboard submissions. Second, we evaluate 3,000 end-to-end repair attempts across 200 SWE-bench instances and three agents (Claude Code, Codex, and the open-source OpenCode) under four execution paradigms, which allows for a fine-grained comparison of performance and cost. Our analysis reveals three key observations: (1) Code execution is used across all agents and models analyzed, with an average of 8.8 test runs per task. Execution behavior varies substantially across agents and models, with frequency ranging from 2 to 19 per task, and late-stage executions consistently achieve higher success rates than early-stage ones. (2) Execution restrictions have little effect on repair success: on commercial agents with SOTA models the resolve-rate gap between Prohibited and Unrestricted is only 1.25 percentage points and not statistically significant, while Prohibited saves substantial token and wall-clock cost. (3) Execution benefit is concentrated rather than uniform. These patterns suggest that current agents apply execution indiscriminately, paying its cost on instances where it provides little benefit. Execution, therefore, should be treated as a resource with an explicit cost-benefit tradeoff, not a default capability.