SWE Refactor Bench：编码智能体能否完成长时程的全仓库技术栈迁移？

摘要

现代软件系统经过数十年的开发积累了大量技术债务，这使得迁移成本高昂且在很大程度上依赖人工。随着编程代理在缺陷修复方面日益强大，它们能否自主执行此类迁移？现有基准测试无法回答这个问题，因为它们只评估行为正确性，而不评估迁移是否实际发生。这导致了一个简单的作弊手段：代理复制原始实现以使测试通过。我们将其称为“盲区”（Blindness）。为解决这一问题，我们提出了SWE Refactor Bench，一个包含20个全仓库迁移的基准测试，涵盖4类技术债务。三阶段评估协议同时衡量迁移完整性和行为正确性：（1）迁移审计（Migration Audit）验证迁移确实发生；（2）行为测试（Behavioural Tests）使用固定测试套件衡量正确性；（3）代理验证（Agentic Verification）使用6个独立的编程代理生成针对性测试，以发现隐藏的行为差异。在来自8个前沿模型和26种模型-努力配置的520次运行中，仅28次（5.4%）通过了全部三个阶段，20个任务中有13个未获得任何可接受方案，最佳模型（claude-opus-5）得分为47.0/100。迁移完整性和行为正确性是两种不同的能力：少数运行通过跳过迁移来保持行为，但在迁移审计阶段被拦截；大多数运行尝试迁移但破坏了行为，并在行为测试阶段被拦截。代理无法交付完美的迁移：在通过迁移审计的340次运行中，58%达到了固定检查项的99%，但仅26%达到100%。代理能力因迁移类别而异：在构建工具链重写上得分为31.4，但在语言重写上仅得5.6。综合来看，这些发现使SWE Refactor Bench成为开发可靠全仓库迁移编程代理的严格测试平台。

English

Modern software systems accumulate technical debt over decades of development, which makes migration expensive and largely manual. As coding agents become increasingly capable at bug fixing, can they autonomously perform such migrations? Existing benchmarks cannot answer this question because they evaluate only behavioural correctness, not whether the migration actually occurred. This leads an easy hack: agents copy the original implementation to make tests pass. We call this Blindness. To address this problem, we introduce SWE Refactor Bench, a benchmark comprising 20 whole-repository migrations, covering 4 kinds of technical debt. A three-stage evaluation protocol measures both migration completeness and behavioural correctness. (1) Migration Audit verifies that the migration occurred. (2) Behavioural Tests measure correctness with a fixed test suite. (3) Agentic Verification uses 6 independent coding agents to generate targeted tests for hidden behavioural differences. Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 of 520 runs (5.4%) pass all three stages, 13 of the 20 tasks receive no accepted solution, and the best model (claude-opus-5) scores 47.0/100. Migration completeness and behavioural correctness are distinct abilities: a few runs preserve behaviour by skipping the migration and are stopped at Migration Audit; most attempt it and break behaviour, and are stopped at Behavioural Tests. Agents cannot deliver a perfect migration: among the 340 runs that pass Migration Audit, 58% reach 99% of the fixed checks, yet only 26% reach 100%. Agent capability differs across migration categories: agents score 31.4 on build toolchain rewrites but only 5.6 on language rewrites. Together, these findings position SWE Refactor Bench as a rigorous testbed for developing coding agents for reliable whole-repository migrations.