运行还是不运行：分析基于大语言模型的程序修复中代码执行的成本效益

摘要

基于大语言模型的程序修复智能体日益采用“生成-运行-修订”范式，通过迭代执行测试来评估和优化补丁。这种基于执行的方法已成为当前最先进系统的标准实践。然而，代码执行可能耗时且昂贵，但其对这些智能体的影响仍未被充分探索。本文通过两阶段实证研究，分析基于大语言模型的程序修复中的执行行为。为大规模刻画执行行为特征，我们首先分析了来自SWE-bench排行榜提交的7,745个智能体追踪记录。其次，我们在200个SWE-bench实例上针对三种智能体（Claude Code、Codex以及开源OpenCode）评估了3,000次端到端修复尝试，并对比了四种执行范式下的性能与成本。分析揭示三项关键发现：（1）所有被分析的智能体和模型均使用代码执行，平均每项任务执行8.8次测试运行。不同智能体和模型的执行行为差异显著，执行频率从每项任务2次到19次不等，且后期执行的成功率始终高于早期执行。（2）执行限制对修复成功率影响甚微：在采用最先进模型的商业智能体上，禁止执行与无限制执行之间的修复率差距仅为1.25个百分点，且无统计学显著性，而禁止执行可大幅节省令牌和运行时间成本。（3）执行收益呈集中分布而非均匀分布。这些模式表明，当前智能体不加区分地使用执行，在收益甚微的实例上仍承担其成本。因此，执行应被视为具有明确成本收益权衡的资源，而非默认能力。

English

LLM-based agents for program repair are increasingly built on a "generate-run-revise" paradigm, iteratively executing tests to evaluate and refine patches. This execution-based approach has become standard practice in state-of-the-art systems. However, executions can be time-consuming and expensive, yet their impact on these agents remains underexplored. In this paper, we conduct a two-stage empirical study over execution behavior in LLM-based program repair. To characterize execution behavior at scale, we first analyze 7,745 agent traces from SWE-bench leaderboard submissions. Second, we evaluate 3,000 end-to-end repair attempts across 200 SWE-bench instances and three agents (Claude Code, Codex, and the open-source OpenCode) under four execution paradigms, which allows for a fine-grained comparison of performance and cost. Our analysis reveals three key observations: (1) Code execution is used across all agents and models analyzed, with an average of 8.8 test runs per task. Execution behavior varies substantially across agents and models, with frequency ranging from 2 to 19 per task, and late-stage executions consistently achieve higher success rates than early-stage ones. (2) Execution restrictions have little effect on repair success: on commercial agents with SOTA models the resolve-rate gap between Prohibited and Unrestricted is only 1.25 percentage points and not statistically significant, while Prohibited saves substantial token and wall-clock cost. (3) Execution benefit is concentrated rather than uniform. These patterns suggest that current agents apply execution indiscriminately, paying its cost on instances where it provides little benefit. Execution, therefore, should be treated as a resource with an explicit cost-benefit tradeoff, not a default capability.