NatureBench：编程代理能否匹配Nature系列期刊论文中已发表的最优结果？

摘要

我們推出 NatureBench，這是一個從同儕審查的《自然》系列出版物中提煉出的跨學科基準測試，包含90項任務，旨在評估AI編碼代理是否能超越單純的複現，真正投入真實科學問題的探索。NatureBench 建構於 NatureGym 之上，這是一套自動化流程，能根據原始論文建立標準化、每項任務各自獨立的容器化環境，解決了長期以來限制先前代理研究基準可信度的環境碎片化問題。在嚴格的「禁止網路搜尋」協議下評估十種前沿代理配置後，我們發現最強的模型在 g>0.1 的標準下僅在 17.8% 的任務中超越了當前最佳水準。方法路徑分析顯示，代理主要透過方法學轉譯取得成功——將科學任務轉化為熟悉的監督式預測問題——而非真正的科學創新。失敗主要歸因於方法選擇錯誤與計算預算不足，而非任務理解錯誤。我們開放了該基準測試、NatureGym 流程，以及一個附維護方再現結果的公開排行榜。程式碼：https://github.com/FrontisAI/NatureBench

English

We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems. NatureBench is built on NatureGym, an automated pipeline that constructs a standardized, per-task containerized environment from a source paper, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. Evaluating ten frontier agent configurations under a strict web-search-disabled protocol, we find that the strongest model surpasses SOTA on only 17.8% of tasks under the g>0.1 criterion. Analysis of method pathways reveals that agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised prediction problems, rather than through genuine scientific invention. Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding. We release the benchmark, the NatureGym pipeline, and a public leaderboard with maintainer-side reproduction. Code: https://github.com/FrontisAI/NatureBench