智能体如何在自动研究中失败：针对100项真实世界前沿研究任务的端到端诊断性评估

摘要

人工智能长期以来一直在辅助科学研究，但大语言模型和智能体框架的快速发展正在重塑这一格局。如今，单一系统即可完成从初始假设到最终论文发表的全流程研究，这种范式现被称为AutoResearch。现有评估对智能体的运作方式及其失效环节揭示甚少：任务范围狭窄，评估仅衡量性能而非过程，失败诊断缺乏系统性覆盖和产物级可见性。为弥补这一空白，我们提出了AutoResearchEval，其包含100项基于已发表前沿科学研究的任务，覆盖7个科学领域和完整的研究生命周期，包括构思、检索、执行、分析、写作和评审。通过评估8种调度框架-模型组合，我们获得了800条自动研究智能体轨迹，并进行了过程级标注。我们将这些见解归纳为AutoResearch失败分类法（ARFT），这是一个包含45种基于实证的失败模式的框架。为实现可扩展的细粒度归因，我们利用经人工校准的“智能体即裁判”流水线来检查完整轨迹和中间产物。失败模式汇聚到一个根本性局限：当前智能体缺乏元认知循环，即能够将自身产物与所获结果进行核对，在这些产物经不起推敲时进行修正，并质疑所采取路径是否合理的能力。这些模式在所有8种调度框架-模型组合中反复出现，包括所测试的最强模型，从而将缺陷定位于模型层面，而非任何特定框架所致。编排层面的干预能否弥补这一缺陷，是一个本工作未检验的开放问题。我们公开发布AutoResearchEval和ARFT，以促进自主科学发现领域的持续研究与发展。

English

AI has long assisted scientific research, but the rapid advance of LLMs and agentic scaffolds is reshaping the landscape; a single system can now carry whole-stage research from an initial hypothesis all the way to final published paper, which is a paradigm now referred to as AutoResearch. Existing evaluations reveal little about how these agents operate or where they break down. Tasks are narrowly-scoped, evaluation measures performance but not process, and failure diagnoses lack systematic coverage or artifact-level visibility. To address this gap, we introduce AutoResearchEval, featuring 100 tasks grounded in published frontier science across 7 scientific domains and the full research lifecycle, including ideation, retrieval, execution, analysis, writing, and review. Evaluating 8 harness-model combinations yields 800 autoresearch agent trajectories, with process-level annotation. We organize these insights into AutoResearch Failure Taxonomy or ARFT, a framework of 45 empirically-grounded failure patterns. To enable scalable fine-grained attribution, we leverage a human-calibrated agent-as-a-judge pipeline to inspect complete trajectories and intermediate artifacts. Failure patterns converge on a single overarching limitation, namely that current agents lack a metacognitive loop, which entails the ability to check what they produced against what they found, revise when it does not hold up, and question whether the path they took was sound. The same patterns recur across all 8 harness-model combinations, including the strongest models tested, locating the deficit at the model level rather than in any particular scaffold; whether orchestration-level interventions can close it is an open question this work does not test. We publicly release AutoResearchEval and ARFT to facilitate continued research and development in autonomous scientific discovery.