下一區塊推理的強化學習真的比監督式微調更好嗎？重新審視無思維鏈數據下的訓練策略

摘要

近期研究提出了下一區塊推理強化學習（next-chunk reasoning RL），用以利用無思維鏈（no-CoT）資料——例如解題步驟與教科書推導等語料庫，這些內容富含推理但缺乏明確的思維鏈標註。該方法訓練模型產生隱式推理軌跡，並依據其預測下一個文字區塊的能力給予獎勵。儘管此方法前景看好，現有評估大多僅與傳統的監督式微調（SFT）基線比較，因此尚未釐清其收益是來自強化學習（RL）框架本身，還是來自更有效地讓模型接觸 no-CoT 資料。我們透過一項針對 next-chunk reasoning RL 的對照研究，並提出一個簡單但先前被忽略的替代方案：混合監督式微調（Mixed SFT），即在單一監督式微調階段中，同時使用 no-CoT 與長思維鏈（long-CoT）資料進行訓練，來回答這個問題。儘管 Mixed SFT 十分簡單，它在 RLVR（可驗證獎勵強化學習）後的效能上限卻明顯高於 next-chunk reasoning RL，且所需訓練運算量少了超過 60 倍。此優勢在域內數學推理與域外推理任務中均保持一致。此外，我們證明 RLVR 前較高的準確度並不一定會轉化為 RLVR 後較高的準確度，這凸顯了在完整後訓練流程的脈絡下評估 no-CoT 訓練策略的必要性。

English

Recent work proposes next-chunk reasoning RL for leveraging no-CoT data---corpora such as worked solutions and textbook derivations that contain reasoning-rich content but lack explicit chain-of-thought annotations. The method trains a model to generate implicit reasoning traces and rewards them by their ability to predict the next chunk of text. While promising, existing evaluations primarily compare against conventional SFT baselines, leaving open whether the gains come from the RL formulation itself or from more effectively exposing the model to no-CoT data. We address this question with a controlled study of next-chunk reasoning RL and a simple but previously overlooked alternative: Mixed SFT, a single supervised fine-tuning stage that jointly trains on no-CoT and long-CoT data. Despite its simplicity, Mixed SFT achieves a clearly higher post-RLVR performance ceiling than next-chunk reasoning RL while requiring over 60 times less training compute. The advantage is consistent across in-domain mathematical reasoning and out-of-domain reasoning tasks. Moreover, we show that higher pre-RLVR accuracy does not necessarily translate into higher post-RLVR accuracy, highlighting the need to evaluate no-CoT training strategies in the context of the full post-training pipeline.