SWE-bench Science：編碼代理能否解決科學領域的工程任務？

摘要

軟體日益成為科學儀器本身的一部分，使得科學程式碼的故障不僅可能損害程式的行為，也可能危及支撐科學結論的證據。然而，現有的編碼代理評估大多側重於整體任務成功率，對於代理在修復科學軟體時為何失敗所提供的洞察相當有限。我們提出 SWE-bench Science，一個針對科學軟體工程的儲存庫層級基準，包含來自 20 個科學領域、98 個 GitHub 儲存庫的 119 項任務。每項任務皆歸類為三種典範之一：問題驅動、專家探索與工程整合。即便是表現最佳的代理——搭載 Opus-5 (max) 的 Claude Code——其 pass@1 仍低於 50%，凸顯了科學軟體工程所帶來的重大挑戰。我們識別出四種反覆出現的失敗機制：科學知識或抽象化能力的不足、探索方向錯誤或僅進行表面修復、修復覆蓋範圍不完整或系統整合不足，以及在分析中未能將科學知識推廣到觀察案例之外。我們進一步進行配對消融實驗，在保留儲存庫與可執行的工程脈絡之同時，移除明確的科學指引。結果顯示，科學知識並非一律有益：有充分根據的資訊能約束修復範圍，並提升平均表現與詞元效率；而對齊不良的指引則可能引發錨定效應，且不一定能提升精確修復的成功率。總而言之，SWE-bench Science 提供了一個廣泛的測試平台，用以研究編碼代理在科學軟體工程中的能力與失敗機制。

English

Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing evaluations of coding agents largely emphasize aggregate task success, providing limited insight into why agents fail when repairing scientific software. We introduce SWE-bench Science, a repository-level benchmark for scientific software engineering comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. Each task is organized into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. Even the best-performing agent, Claude Code with Opus-5 (max), achieves a pass@1 below 50\%, highlighting the substantial challenges posed by scientific software engineering. We identify four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases in our analysis. We further conduct a paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context. The results show that scientific knowledge is not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success. Together, SWE-bench Science provides a broad testbed for studying both the capabilities and failure mechanisms of coding agents in scientific software engineering.