迈向基于谷歌论文助手工具的科学评审自动化

摘要

人工智能正在推动科学发现的革命，加速从假设生成到数学定理证明的各个环节。然而，这种快速加速也带来系统性挑战：传统的人类同行评审无法匹配人工智能辅助科学成果的涌入速度。最终，要解决这一矛盾，我们必须同时部署人工智能来加速验证和评审过程本身。为构建这一转型的讨论框架，我们提出了一种包含人工智能与人类在科学评估中四种渐进式协作水平的分类体系，并探讨了每种水平涉及的各种权衡。作为迈向这一未来的举措，我们推出了论文辅助工具（PAT）——一个专为深度科学评审与验证而构建的智能体AI框架。PAT可读取完整科学手稿并生成全面评估，包括检验理论结果、验证实验、提出改进建议及识别潜在缺陷。通过利用推理缩放技术，PAT能够识别单次模型调用无法发现的深层问题，在SPOT基准测试中对数学错误的零样本召回率提升了34%。PAT作为投稿前工具，已在两大计算机科学顶级会议——STOC和ICML——的作者群体中开展试点部署，证明其能够识别关键错误并为研究论文提出实质性改进建议。通过及早发现错误，PAT减轻了审稿人的认知负担，同时保留了他们对评审结果的控制权。

English

Artificial intelligence is driving a revolution in scientific discovery, accelerating everything from hypothesis generation to mathematical theorem proving. However, this rapid acceleration is creating a systemic challenge: traditional human peer review cannot scale to match the influx of AI-assisted science. Ultimately, to resolve this tension, we must also deploy AI to accelerate the verification and review process itself. To frame the discussion around this transition, we propose a taxonomy consisting of four progressive levels of AI-human collaboration in scientific evaluation, and discuss various trade-offs involved with each. As a step toward this future, we introduce the Paper Assistant Tool (PAT), an agentic AI framework built for deep scientific review and verification. PAT ingests full scientific manuscripts and produces a comprehensive evaluation, checking theoretical results, validating experiments, suggesting improvements, and identifying potential flaws. By utilizing inference scaling techniques, PAT is able to identify deeper issues than a single model call alone, achieving a 34% improvement over zero-shot recall on mathematical errors in the SPOT benchmark. Pilot deployments of PAT as a pre-submission tool for authors at two major Computer Science conferences -- STOC and ICML -- demonstrate its ability to identify critical errors and suggest substantive improvements to research papers. By catching errors early, PAT eases the cognitive burden placed on referees, while preserving their control over the outcomes of the review process.