PaperGym：以评分标准为中心的进化式研究计划生成

摘要

研究规划是AI科学家的决定性能力。然而，研究规划不存在可验证的答案，因此强化学习缺乏其所需的环境：任务与评价器的配对。从科学论文中提取的评分标准可以提供评价器。然而，现有流程从相同内容中提取问题和评分标准，因此可以通过释义获得奖励。评分标准进一步被压缩为每次轨迹的单一标量。我们提出PaperGym，一个将每篇研究论文转化为完整训练环境的统一框架。PaperGym利用论文的结构：问题从研究目标和背景中综合生成，而评分标准则从方法和实验中推导得出。评分标准涵盖方法论创新和实验设计，标准泄漏率降至3.7%，而现有数据集为11.90%至34.10%。训练过程中评分标准被使用两次：首先作为OPSD自教师模型的特权上下文，然后作为GRPO的奖励。在Qwen3-1.7B/4B/8B上，该训练方案优于监督微调、任一单独阶段以及相反顺序，将五基准平均值分别提升+5.6、+5.0和+4.8个百分点。在训练方案保持不变的情况下，基于PaperGym-20k训练的模型在三方比较中获胜58.1%，而RubricHub Science为28.2%。训练后的Qwen3-8B在ResearchQA上达到73.48，超过了规模大得多的Kimi K2.6。我们发布了该流程、包含20,000个实例的语料库PaperGym-20k，以及基准PaperGym-Innov和PaperGym-Design。

English

Research planning is the decisive capability of AI scientists. Yet a research plan admits no verifiable answer, so reinforcement learning lacks the environment it requires: tasks paired with a critic. Rubrics extracted from scientific papers can supply the critic. Existing pipelines, however, draw the question and the criteria from the same content, so the reward can be earned by paraphrase. The rubric is further compressed into a single scalar per rollout. We introduce PaperGym, a unified framework that turns each research paper into a complete training environment. PaperGym exploits the structure of a paper: the question is synthesized from the research goal and background, while the criteria are derived from the method and experiments. The criteria span methodological innovation and experimental design, and criterion leakage falls to 3.7%, versus 11.90% to 34.10% in existing datasets. Training uses the rubric twice: first as privileged context for OPSD's self-teacher, then as the reward for GRPO. Across Qwen3-1.7B/4B/8B, this schedule outperforms supervised fine-tuning, either stage alone, and the reverse ordering, improving five-benchmark averages by +5.6, +5.0, and +4.8 points. With the recipe held fixed, models trained on PaperGym-20k win 58.1% of three-way comparisons, against 28.2% for RubricHub Science. The trained Qwen3-8B reaches 73.48 on ResearchQA, above the far larger Kimi K2.6. We release the pipeline, the 20,000-instance corpus PaperGym-20k, and the benchmarks PaperGym-Innov and PaperGym-Design.