從推理鏈到可驗證的子問題：課程強化學習賦能大型語言模型推理的信用分配

摘要

基于可验证奖励的强化学习（RLVR）在提升大语言模型推理能力方面展现出巨大潜力，但基于结果驱动的RLVR在处理难题时效率依然低下，原因在于正确最终答案的生成过程罕见，且样本层面的信用分配无法利用失败尝试中的部分进展。为此，我们提出了SCRL（子问题课程强化学习），这是一种课程式强化学习框架，通过从参考推理链中提取可验证的子问题，并将最终子问题固定为原始问题。这一设计将难题的部分进展转化为可验证的学习信号。在算法层面，SCRL采用子问题级归一化，即针对每个子问题位置独立地归一化奖励，并将由此产生的优势值分配给相应的答案片段，从而无需外部评分标准或奖励模型即可实现更细粒度的信用分配。我们的分析表明，子问题课程能够将难题从梯度死区中解放出来，且原始问题越难，相对增益越显著。在七个数学推理基准测试中，SCRL的表现优于强课程学习基线：在Qwen3-4B-Base上，平均准确率比GRPO高出4.1个百分点；在Qwen3-14B-Base上则高出1.9个百分点。在AIME24、AIME25和IMO-Bench上，SCRL进一步将Qwen3-4B-Base的pass@1提升了3.7个百分点，pass@64提升了4.6个百分点，表明其在复杂推理问题上具有更优的探索能力。

English

Reinforcement learning from verifiable rewards (RLVR) has shown strong promise for LLM reasoning, but outcome-based RLVR remains inefficient on hard problems because correct final-answer rollouts are rare and sample-level credit assignment cannot use partial progress in failed attempts. We introduce SCRL (Subproblem Curriculum Reinforcement Learning), a curriculum RL framework that derives verifiable subproblems from reference reasoning chains and fixes the final subproblem as the original problem. This turns partial progress on hard problems into verifiable learning signals. Algorithmically, SCRL uses subproblem-level normalization, which normalizes rewards independently at each subproblem position and assigns the resulting advantages to the corresponding answer spans, enabling finer-grained credit assignment without external rubrics or reward models. Our analysis shows that subproblem curricula lift hard problems out of gradient dead zones, with larger relative gains as the original problem becomes harder. Across seven mathematical reasoning benchmarks, SCRL outperforms strong curriculum-learning baselines, improving average accuracy over GRPO by +4.1 points on Qwen3-4B-Base and +1.9 points on Qwen3-14B-Base. On AIME24, AIME25, and IMO-Bench, SCRL further improves pass@1 by +3.7 points and pass@64 by +4.6 points on Qwen3-4B-Base, indicating better exploration on hard reasoning problems.