SWE-bench Science：编码智能体能解决科学领域的工程任务吗？

摘要

软件日益成为科学仪器本身的一部分，这使得科学代码中的故障不仅可能损害程序行为，还可能危及支撑科学结论的证据。然而，现有对编码智能体的评估大多侧重于整体任务成功率，对于智能体在修复科学软件时为何失败所提供的洞察十分有限。我们提出 SWE-bench Science——一个面向科学软件工程的仓库级基准，包含来自 20 个科学领域、98 个 GitHub 仓库中的 119 个任务。每个任务归属于三种范式之一：问题驱动型、专家探索型和工程集成型。即使性能最优的智能体 Claude Code with Opus-5 (max)，其 pass@1 也低于 50%，凸显了科学软件工程所带来的巨大挑战。我们识别出四种反复出现的失败机制：科学知识或抽象能力的欠缺、误导性探索或浅层修复、修复覆盖或系统集成不完整，以及在我们的分析中未能将科学知识推广到所观察案例之外的失败。我们进一步进行了配对消融实验，在保留仓库和可执行工程上下文的同时移除显式科学指导。结果表明，科学知识并非 uniformly 有益：充分依据的信息可以约束修复过程并提高平均性能和 Token 效率，而契合度欠佳的指导则可能引发锚定效应，且未必能提升精确修复成功率。综上，SWE-bench Science 为研究编码智能体在科学软件工程中的能力与失败机制提供了一个广泛的测试平台。

English

Software increasingly functions as part of the scientific instrument itself, making failures in scientific code capable of compromising not only program behavior but also the evidence underlying scientific conclusions. Yet existing evaluations of coding agents largely emphasize aggregate task success, providing limited insight into why agents fail when repairing scientific software. We introduce SWE-bench Science, a repository-level benchmark for scientific software engineering comprising 119 tasks from 98 GitHub repositories across 20 scientific domains. Each task is organized into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. Even the best-performing agent, Claude Code with Opus-5 (max), achieves a pass@1 below 50\%, highlighting the substantial challenges posed by scientific software engineering. We identify four recurring failure mechanisms: deficits in scientific knowledge or abstraction, misguided exploration or surface-level repair, incomplete repair coverage or system integration, and failures to generalize scientific knowledge beyond observed cases in our analysis. We further conduct a paired ablation that removes explicit scientific guidance while preserving the repository and executable engineering context. The results show that scientific knowledge is not uniformly beneficial: well-grounded information can constrain repair and improve average performance and token efficiency, whereas poorly aligned guidance can induce anchoring and does not necessarily improve exact repair success. Together, SWE-bench Science provides a broad testbed for studying both the capabilities and failure mechanisms of coding agents in scientific software engineering.