基于持久工作流提示、元提示与元推理的AI驱动学术同行评审

AI-Driven Scholarly Peer Review via Persistent Workflow Prompting, Meta-Prompting, and Meta-Reasoning

May 6, 2025

作者: Evgeny Markhasin

cs.AI

摘要

科学论文的同行评审对大型语言模型（LLMs）构成了重大挑战，部分原因在于数据限制及专家推理的复杂性。本报告介绍了持久工作流提示（Persistent Workflow Prompting, PWP），这是一种可能广泛适用的提示工程方法，旨在利用标准LLM聊天界面（零代码、无需API）来弥合这一差距。我们展示了一个用于实验化学论文批判性分析的概念验证PWP提示，该提示采用层次化、模块化架构（通过Markdown结构化），定义了详细的分析工作流。我们通过迭代应用元提示技术和元推理，开发了这一PWP提示，旨在系统化编码专家评审工作流，包括隐性知识。在会话开始时提交一次，此PWP提示为LLM配备了持久工作流，由后续查询触发，引导现代推理LLM进行系统化、多模态评估。演示表明，PWP引导的LLM在测试案例中识别出主要方法缺陷，同时减轻了LLM输入偏差，并执行了复杂任务，包括区分主张与证据、整合文本/照片/图表分析以推断参数、执行定量可行性检查、将估计值与主张对比，以及评估先验合理性。为确保透明度并促进复现，我们提供了完整提示、详细演示分析及互动聊天日志作为补充资源。除具体应用外，本工作还深入探讨了元开发过程本身，强调了PWP在详细工作流形式化指导下，利用现成LLM进行复杂科学任务高级分析的潜力。

English

Critical peer review of scientific manuscripts presents a significant challenge for Large Language Models (LLMs), partly due to data limitations and the complexity of expert reasoning. This report introduces Persistent Workflow Prompting (PWP), a potentially broadly applicable prompt engineering methodology designed to bridge this gap using standard LLM chat interfaces (zero-code, no APIs). We present a proof-of-concept PWP prompt for the critical analysis of experimental chemistry manuscripts, featuring a hierarchical, modular architecture (structured via Markdown) that defines detailed analysis workflows. We develop this PWP prompt through iterative application of meta-prompting techniques and meta-reasoning aimed at systematically codifying expert review workflows, including tacit knowledge. Submitted once at the start of a session, this PWP prompt equips the LLM with persistent workflows triggered by subsequent queries, guiding modern reasoning LLMs through systematic, multimodal evaluations. Demonstrations show the PWP-guided LLM identifying major methodological flaws in a test case while mitigating LLM input bias and performing complex tasks, including distinguishing claims from evidence, integrating text/photo/figure analysis to infer parameters, executing quantitative feasibility checks, comparing estimates against claims, and assessing a priori plausibility. To ensure transparency and facilitate replication, we provide full prompts, detailed demonstration analyses, and logs of interactive chats as supplementary resources. Beyond the specific application, this work offers insights into the meta-development process itself, highlighting the potential of PWP, informed by detailed workflow formalization, to enable sophisticated analysis using readily available LLMs for complex scientific tasks.