SWE-Bench ProMax：大規模多語言程式碼重構之智能體基準測試

摘要

隨著 AI 程式設計代理承擔越來越複雜、長程的軟體工程任務，現有基準正迅速飽和，其評估品質也受到嚴格檢視：近期一項審查發現，近 60% 未解決的 SWE-bench Verified 實例含有有缺陷的測試——這些測試可能過於狹窄，拒絕了正確的解決方案；也可能過於寬泛，檢查了未陳述的需求——且前沿模型可以逐字重現訓練資料中的黃金修補程式。程式碼重構需要在多個檔案之間進行協調且保持行為不變的變更，這對代理能力構成了難度高得多且更貼近現實的測試，然而現有基準對此領域的涵蓋仍嚴重不足。我們提出了 SWE-Bench ProMax，這是一個由專家策劃、多語言的程式碼重構基準，包含 170 個實例，取自七種程式語言（Python、Java、TypeScript、Go、C、C++ 和 Rust）的真實提交。每個實例都經過嚴謹的多階段篩選，直接解決了先前基準中識別出的品質問題：問題描述從頭重寫，以提供精確且無歧義的規格；測試套件經人工審查，以移除過於狹窄和過於寬泛的測試。複雜度不足或跨檔案範圍有限的任務會被篩除，最終形成一個具挑戰性的大規模重構任務基準，平均每個實例修改 11.4 個檔案和 261.6 行程式碼，規模大幅超過現有基準。我們在兩種代理框架下對前沿模型進行的實驗顯示，最佳模型僅達到 41.2% 的解決率，這證實 SWE-Bench ProMax 對當前 AI 程式設計代理構成了一個重要且尚未飽和的挑戰。我們的基準可於 https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax 取得。

English

As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.4 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents. Our benchmark is available at https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.