理解从预训练到后训练的推理

摘要

强化学习（RL）已成为提升大语言模型（LLMs）在复杂推理任务上表现的核心技术，然而当前对RL后训练的研究大多独立于其前的预训练阶段。由此导致两个基本问题悬而未决：（1）预训练选择（模型规模、数据）如何影响RL计算投入的回报率？（2）RL究竟对模型产生了什么影响？这些问题在标准LLM设置下难以研究：预训练语料库规模庞大且无法受控，导致难以将模型行为归因于预训练还是RL，且对两个阶段进行系统性的计算量扫描成本过高。为解决这些挑战，我们以国际象棋为受控实验平台，研究从预训练到后训练全流程的推理能力。我们遵循标准LLM训练流程：首先在大规模人类棋谱上预训练参数规模从500万到10亿不等的语言模型，随后在合成推理轨迹上进行监督微调，最后对带有可验证奖励的国际象棋谜题实施RL训练。利用这一框架，我们发现：在给定RL计算量下，后RL性能可通过预训练损失进行良好预测，且RL奖励曲线的斜率随预训练数据量近似线性提升。除规模规律外，我们还发现RL并非简单地对监督微调（SFT）策略进行锐化：在简单谜题上，它放大了SFT策略原本就偏好的正确走法；而在困难谜题上，它发掘出了SFT策略几乎从未出现的正确走法。我们进一步验证这些发现是否超越国际象棋领域，通过在数学领域文本上训练10亿参数语言模型，观察到相同的预测模式：预训练更充分的检查点在RL后达到更高性能，且在RL训练中提升更快。总之，我们提供了预训练与RL接口的定量描述，以及一个可受控的测试平台，用于研究从预训练到后训练全流程的推理科学。

English

Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.