在改进之前先学习评估：为自动化研究智能体自动归纳评估标准

摘要

自主科学研究智能体日益被应用于端到端的科学工作流程，涵盖文献综述、数据分析、实验和报告生成。然而，开放式研究任务往往并未明确指定完成任务所需的分析、方法和成功标准。因此，智能体可能遗漏重要分析、采用不恰当的方法，或得出缺乏充分证据支持的结论。针对这一问题，我们提出了AutoSciRub，一个评估优先的框架，它在研究执行之前生成特定任务的可执行评估准则，并利用该准则指导执行、准则级验证以及迭代修订。AutoSciRub将表述不清的指令分解为原子化的科学目标，将其锚定于相关文献和任务可见数据，并综合出具体、可操作且可验证的标准。由此产生的评估准则使隐含的实验和证据要求显式化，为实验和分析提供指引。在修订过程中，准则引导的验证能够识别未满足的标准，从而实现对研究报告及其支持性产物的定向改进。在ResearchClawBench上，AutoSciRub一致提升了所有测试配置，在固定Codex执行框架下，三个骨干大语言模型的平均提升为2.08分；在固定使用DeepSeek-V4-Flash骨干模型的情况下，三个智能体执行框架的平均提升为2.95分。在AstaBench E2E Discovery随机抽样的20个任务子集上，AutoSciRub在三个智能体执行框架上进一步实现了平均16.8分的提升，同时保持或增加了成功完成的任务数量。这些结果表明，评估优先的指导为自主科学研究提供了一种有效且可泛化的控制机制（代码：https://github.com/zjunlp/AutoSciRub）。

English

Autonomous scientific research agents are increasingly applied to end-to-end scientific workflows, including literature review, data analysis, experimentation, and report generation. However, open-ended research tasks often do not clearly specify the analyses, methods, and success criteria required to complete the task. As a result, agents may miss important analyses, use inappropriate methods, or draw conclusions that are insufficiently supported by evidence. To address the problem, we present AutoSciRub, an evaluation-first framework that induces a task-specific executable rubric before research execution, and uses it to guide execution, criterion-level verification as well as iterative revision. AutoSciRub decomposes an underspecified instruction into atomic scientific goals, grounds them in relevant literature and task-visible data, and synthesizes specific, actionable, and verifiable criteria. The resulting rubric makes implicit experimental and evidential requirements explicit, providing guidance for experiments and analyses. During revision, rubric-guided verification identifies unmet criteria and enables targeted refinement of the research report and its supporting artifacts. On ResearchClawBench, AutoSciRub consistently improves all tested configurations, with an average gain of 2.08 points across three backbone LLMs under the fixed Codex harness and 2.95 points across three agent harnesses using a fixed DeepSeek-V4-Flash backbone. On a randomly sampled 20-task subset of AstaBench E2E Discovery, AutoSciRub further achieves an average improvement of 16.8 points across three agent harnesses, while maintaining or increasing the number of successfully completed tasks. These results demonstrate that evaluation-first guidance provides an effective and generalizable control mechanism for autonomous scientific research (Code: https://github.com/zjunlp/AutoSciRub).