從預訓練到後訓練理解推理

摘要

強化學習（Reinforcement Learning, RL）已成為改善大型語言模型（Large Language Models, LLMs）在複雜推理任務上表現的核心方法，然而，RL 的後訓練階段大多與其之前的預訓練階段分開研究。因此，兩個基本問題仍未解決：（1）預訓練的選擇（模型大小、數據）如何影響 RL 運算的效益回報，以及（2）RL 實際對模型做了什麼？這些問題在標準的大型語言模型設定中難以研究：預訓練語料庫龐大且不受控，使得難以將行為歸因於預訓練或 RL，而對兩個階段進行系統性的運算量掃描則成本過高。為了解決這些挑戰，我們使用西洋棋作為一個受控的測試平台，來研究從預訓練到後訓練的完整流程中的推理能力。我們遵循標準的大型語言模型訓練流程：先在人際對弈的西洋棋棋譜上預訓練參數量從 5M 到 1B 不等的語言模型，接著在合成推理軌跡上進行監督式微調（Supervised Fine-Tuning, SFT），最後對具有可驗證獎勵的西洋棋謎題執行 RL。透過這個框架，我們發現：在給定的 RL 運算量下，RL 後的表現可以透過預訓練損失來良好預測，而 RL 獎勵曲線的斜率大致上與預訓練的 token 數呈線性增長。除了規模效應之外，我們還發現 RL 並非僅僅強化 SFT 策略：在簡單的謎題上，它放大了 SFT 策略原本就已偏好的正確走法；而在困難的謎題上，它則浮現出 SFT 下幾乎不存在的正確走法。我們進一步測試這些發現是否能夠遷移至西洋棋以外的領域，方法是在數學領域文本上訓練一個 1B 參數量的語言模型，結果出現了相同的預測模式：預訓練時間更長的檢查點能達到更高的 RL 後表現，並且在 RL 下改善速度更快。總而言之，我們提供了預訓練與 RL 之間介面的量化描述，以及一個用於研究從預訓練到後訓練完整流程中推理科學的受控測試平台。

English

Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we find that the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens. Beyond scaling, we find that RL does not simply sharpen the SFT policy: on easy puzzles it amplifies correct moves the SFT policy already preferred, while on hard puzzles it surfaces correct moves that were nearly absent under SFT. We further test whether our findings transfer beyond chess by training a 1B language model on math-domain text, where the same predictive pattern emerges: longer-pretrained checkpoints reach higher post-RL performance and improve faster under RL. In sum, we provide a quantitative account of the pretraining-to-RL interface and a controlled testbed for studying the science of reasoning across the full pretraining-to-post-training pipeline.