SWE 重構基準：編碼代理能否完成長程、全儲存庫的技術棧遷移？

摘要

現代軟體系統在數十年的開發過程中累積了大量技術債，使得遷移成本高昂且大部分依賴人工。隨著編碼代理在錯誤修復方面日益強大，它們能否自主執行此類遷移？現有基準測試無法回答這個問題，因為它們只評估行為正確性，而不評估遷移是否實際發生。這導致了一個容易的投機取巧方式：代理複製原始實作以讓測試通過。我們將此稱為盲目性（Blindness）。為了解決這個問題，我們提出了 SWE Refactor Bench，一個包含 20 個全倉庫遷移、涵蓋 4 種類型技術債的基準測試。三階段評估協議同時衡量遷移完整性與行為正確性。(1) 遷移審計（Migration Audit）驗證遷移是否確實發生。(2) 行為測試（Behavioural Tests）使用固定測試套件衡量正確性。(3) 代理式驗證（Agentic Verification）使用 6 個獨立編碼代理為隱藏的行為差異生成有針對性的測試。在來自 8 個前沿模型和 26 種模型-努力配置的 520 次運行中，僅有 28 次運行（5.4%）通過全部三個階段，20 個任務中有 13 個未獲得任何被接受的解決方案，最佳模型（claude-opus-5）得分為 47.0/100。遷移完整性與行為正確性是兩種不同的能力：少數運行透過跳過遷移來保持行為，在遷移審計階段被攔截；大多數運行嘗試遷移但破壞了行為，在行為測試階段被攔截。代理無法交付完美的遷移：在通過遷移審計的 340 次運行中，58% 達到了固定檢查的 99%，但僅有 26% 達到 100%。代理能力因遷移類別而異：代理在建置工具鏈重寫上得分 31.4，但在語言重寫上僅得分 5.6。綜合這些發現，SWE Refactor Bench 可作為開發可靠全倉庫遷移編碼代理的嚴謹測試平台。

English

Modern software systems accumulate technical debt over decades of development, which makes migration expensive and largely manual. As coding agents become increasingly capable at bug fixing, can they autonomously perform such migrations? Existing benchmarks cannot answer this question because they evaluate only behavioural correctness, not whether the migration actually occurred. This leads an easy hack: agents copy the original implementation to make tests pass. We call this Blindness. To address this problem, we introduce SWE Refactor Bench, a benchmark comprising 20 whole-repository migrations, covering 4 kinds of technical debt. A three-stage evaluation protocol measures both migration completeness and behavioural correctness. (1) Migration Audit verifies that the migration occurred. (2) Behavioural Tests measure correctness with a fixed test suite. (3) Agentic Verification uses 6 independent coding agents to generate targeted tests for hidden behavioural differences. Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 of 520 runs (5.4%) pass all three stages, 13 of the 20 tasks receive no accepted solution, and the best model (claude-opus-5) scores 47.0/100. Migration completeness and behavioural correctness are distinct abilities: a few runs preserve behaviour by skipping the migration and are stopped at Migration Audit; most attempt it and break behaviour, and are stopped at Behavioural Tests. Agents cannot deliver a perfect migration: among the 340 runs that pass Migration Audit, 58% reach 99% of the fixed checks, yet only 26% reach 100%. Agent capability differs across migration categories: agents score 31.4 on build toolchain rewrites but only 5.6 on language rewrites. Together, these findings position SWE Refactor Bench as a rigorous testbed for developing coding agents for reliable whole-repository migrations.