基于科学家工作流对齐的大语言模型科学通用智能测评

摘要

尽管科学人工智能取得了进展，但科学通用智能仍缺乏统一框架——这种能够自主构思、探索和跨领域推理的能力尚未形成体系。我们提出基于实践探究模型的操作性定义，并通过四个与科学家工作对齐的任务实现其操作化：深度研究、创意生成、干湿实验及实验推理。受《科学》杂志125个重大议题启发构建的SGI-Bench包含千余个专家精选的跨学科样本，可系统评估前沿大语言模型。研究揭示多重差距：深度研究虽具步骤匹配性但精确匹配率低；创意缺乏可行性与细节；干实验代码可执行性高但结果准确率低；湿实验流程序列保真度不足；多模态比较推理挑战持续存在。我们进一步提出测试时强化学习技术，通过在推理阶段优化检索增强的新颖性奖励，在无参考答案情况下提升假设创新性。基于实践探究模型的定义、以工作流为核心的基准测试及实证发现，共同为真正参与科学发现的人工智能系统奠定了基础。

English

Despite advances in scientific AI, a coherent framework for Scientific General Intelligence (SGI)-the ability to autonomously conceive, investigate, and reason across scientific domains-remains lacking. We present an operational SGI definition grounded in the Practical Inquiry Model (PIM: Deliberation, Conception, Action, Perception) and operationalize it via four scientist-aligned tasks: deep research, idea generation, dry/wet experiments, and experimental reasoning. SGI-Bench comprises over 1,000 expert-curated, cross-disciplinary samples inspired by Science's 125 Big Questions, enabling systematic evaluation of state-of-the-art LLMs. Results reveal gaps: low exact match (10--20%) in deep research despite step-level alignment; ideas lacking feasibility and detail; high code executability but low execution result accuracy in dry experiments; low sequence fidelity in wet protocols; and persistent multimodal comparative-reasoning challenges. We further introduce Test-Time Reinforcement Learning (TTRL), which optimizes retrieval-augmented novelty rewards at inference, enhancing hypothesis novelty without reference answer. Together, our PIM-grounded definition, workflow-centric benchmark, and empirical insights establish a foundation for AI systems that genuinely participate in scientific discovery.

基于科学家工作流对齐的大语言模型科学通用智能测评

Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows

摘要

Support