PaperGym：以評分標準為中心的演化式研究計畫生成

摘要

研究規劃是 AI 科學家的決定性能力。然而，研究計畫沒有可驗證的答案，因此強化學習缺乏其所需的環境：與評判器配對的任務。從科學論文中提取的評分標準可以提供評判器。然而，現有流程從相同內容中提取問題與標準，因此可藉由改寫獲得獎勵。評分標準更被壓縮為每次 rollout 的單一標量。我們提出 PaperGym，一個將每篇研究論文轉化為完整訓練環境的統一框架。PaperGym 利用論文結構：問題綜合自研究目標與背景，而標準則源自方法與實驗。這些標準涵蓋方法創新與實驗設計，評分標準洩漏率降至 3.7%，而現有資料集則介於 11.90% 至 34.10% 之間。訓練兩次使用評分標準：首先作為 OPSD 自我教師的特權上下文，然後作為 GRPO 的獎勵。在 Qwen3-1.7B/4B/8B 上，此排程優於監督式微調、單獨任一個階段以及相反的順序，將五個基準的平均值分別提高 +5.6、+5.0 和 +4.8 個百分點。在配方固定的情況下，使用 PaperGym-20k 訓練的模型在三方比較中勝率為 58.1%，而 RubricHub Science 僅為 28.2%。訓練後的 Qwen3-8B 在 ResearchQA 上達到 73.48，超過規模大得多的 Kimi K2.6。我們釋出流程、包含 20,000 個實例的語料庫 PaperGym-20k，以及基準 PaperGym-Innov 和 PaperGym-Design。

English

Research planning is the decisive capability of AI scientists. Yet a research plan admits no verifiable answer, so reinforcement learning lacks the environment it requires: tasks paired with a critic. Rubrics extracted from scientific papers can supply the critic. Existing pipelines, however, draw the question and the criteria from the same content, so the reward can be earned by paraphrase. The rubric is further compressed into a single scalar per rollout. We introduce PaperGym, a unified framework that turns each research paper into a complete training environment. PaperGym exploits the structure of a paper: the question is synthesized from the research goal and background, while the criteria are derived from the method and experiments. The criteria span methodological innovation and experimental design, and criterion leakage falls to 3.7%, versus 11.90% to 34.10% in existing datasets. Training uses the rubric twice: first as privileged context for OPSD's self-teacher, then as the reward for GRPO. Across Qwen3-1.7B/4B/8B, this schedule outperforms supervised fine-tuning, either stage alone, and the reverse ordering, improving five-benchmark averages by +5.6, +5.0, and +4.8 points. With the recipe held fixed, models trained on PaperGym-20k win 58.1% of three-way comparisons, against 28.2% for RubricHub Science. The trained Qwen3-8B reaches 73.48 on ResearchQA, above the far larger Kimi K2.6. We release the pipeline, the 20,000-instance corpus PaperGym-20k, and the benchmarks PaperGym-Innov and PaperGym-Design.