下一分块推理的强化学习真的优于监督微调吗？——重新审视无思维链数据下的训练策略

摘要

近期工作提出了下一分块推理强化学习（next-chunk reasoning RL），用于利用无CoT数据——即包含丰富推理内容但缺乏显式思维链标注的语料库，例如解题过程和教材推导。该方法训练模型生成隐式推理轨迹，并以其预测下一分块文本的能力作为奖励。尽管这一方法前景可观，现有评估主要对比传统有监督微调（SFT）基线，因此尚不清楚其收益究竟来自强化学习本身，还是来自更有效地让模型接触无CoT数据。我们通过一项受控研究来回答这一问题，对比了下一分块推理强化学习与一个简单但此前被忽视的替代方案：混合SFT（Mixed SFT），即在单一有监督微调阶段中联合训练无CoT数据与长CoT数据。尽管方法简单，混合SFT在RLVR后的性能上限明显高于下一分块推理强化学习，同时所需训练算力不到后者的1/60。该优势在领域内数学推理和领域外推理任务上均保持一致。此外，我们表明RLVR前的高准确率并不必然转化为RLVR后的高准确率，这凸显了在完整后训练流程背景下评估无CoT训练策略的必要性。

English

Recent work proposes next-chunk reasoning RL for leveraging no-CoT data---corpora such as worked solutions and textbook derivations that contain reasoning-rich content but lack explicit chain-of-thought annotations. The method trains a model to generate implicit reasoning traces and rewards them by their ability to predict the next chunk of text. While promising, existing evaluations primarily compare against conventional SFT baselines, leaving open whether the gains come from the RL formulation itself or from more effectively exposing the model to no-CoT data. We address this question with a controlled study of next-chunk reasoning RL and a simple but previously overlooked alternative: Mixed SFT, a single supervised fine-tuning stage that jointly trains on no-CoT and long-CoT data. Despite its simplicity, Mixed SFT achieves a clearly higher post-RLVR performance ceiling than next-chunk reasoning RL while requiring over 60 times less training compute. The advantage is consistent across in-domain mathematical reasoning and out-of-domain reasoning tasks. Moreover, we show that higher pre-RLVR accuracy does not necessarily translate into higher post-RLVR accuracy, highlighting the need to evaluate no-CoT training strategies in the context of the full post-training pipeline.