性能优化基准能否可靠地评估编码智能体？

摘要

仓库级别的性能优化基准测试（如GSO、SWE-Perf和SWE-fficiency）通过将补丁应用于真实仓库，并将运行时间与未优化的基线和官方参考补丁进行对比，来评估编码代理。其排行榜分数越来越多地被用作编码代理进展的证据，但这些分数可能混淆运行时的不稳定性、特定基准测试的评分规则，以及已有多个任务被至少一个公开提交解决的情况。我们对这三个基准测试进行了审计。首先，我们在四种常见的谷歌云机器上重放了740个代码优化任务的官方参考补丁。大多数基准测试任务可以重放，但只有39/102个GSO任务、11/140个SWE-Perf任务和411/498个SWE-fficiency任务满足原始基准测试的有效性规则（在所有跨机器重放中）；SWE-Perf尤其脆弱，因为许多参考补丁产生的运行时间变化接近零。其次，我们表明公开提交的排名强烈依赖于基准测试评分规则。在GSO和SWE-fficiency共享的八个公开提交中，官方排名在28个两两提交比较中有9个存在分歧，并且SWE-fficiency的排行榜评分规则将最差的十个任务分配了过高的分数权重（58.5%-82.8%）。第三，通过分析每个任务的十个公开提交，我们发现对于85.3%（384/450）的可重放GSO和SWE-fficiency任务，至少有一个提交匹配或超过了参考补丁；在99.8%（449/450）的任务中，至少有一个提交超过了未优化的基线代码。我们的研究通过识别具有更可靠性能信号的任务、量化每个任务的分数贡献、以及揭示聚合排名中隐藏的剩余性能差距，补充了排行榜分数。

English

Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, we replay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%-82.8%. Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.