性能優化基準是否可靠地評估編碼代理？

摘要

在仓库级别的性能优化基准测试中，如 GSO、SWE-Perf 和 SWE-fficiency，通过将补丁应用于真实仓库，并将运行时与未优化基线及官方参考补丁进行对比，来评估编码智能体的性能。这些基准的排行榜得分越来越多地被用作编码智能体进展的证据，但这类得分可能混淆运行时的不稳定性、基准特定的评分规则，以及有多少任务已被至少一个公开提交所解决。我们对这三个基准中的这些问题进行了核查。首先，我们在四种常见的谷歌云机器上重放了 740 个代码优化任务的官方参考补丁。大多数基准任务可以重放，但在跨机器重放中，其参考补丁满足原始基准有效性规则的任务数量分别为：GSO 中 102 个任务仅有 39 个，SWE-Perf 中 140 个任务仅有 11 个，SWE-fficiency 中 498 个任务仅有 411 个；SWE-Perf 尤其脆弱，因为许多参考补丁产生的运行时变化接近零。其次，我们证明公开提交的排名在很大程度上取决于基准的评分规则。在 GSO 和 SWE-fficiency 共享的八个公开提交中，官方排名在 28 组两两提交比较中有 9 组存在分歧，而 SWE-fficiency 的排行榜评分规则将最差的十个任务赋予过高的得分权重（58.5%-82.8%）。第三，观察每个任务的十个公开提交，我们发现至少有一个提交在 85.3%（384/450）的可重放有效的 GSO 和 SWE-fficiency 任务中匹配或超越了参考补丁，并在 99.8%（449/450）的任务中超越了未优化的基础代码。我们的研究通过识别具有更可靠性能信号的任务、量化每项任务的得分贡献，以及揭示被汇总排名所掩盖的剩余性能差距，对排行榜得分进行了补充。

English

Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, we replay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%-82.8%. Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.