Cliff：从首个错误中学习过程奖励

摘要

基于可验证奖励的强化学习（RLVR）已成为大语言模型（LLM）后训练的一种强大范式，但其对粗粒度结果奖励的依赖导致对中间推理过程的指导十分有限。现有方法如过程奖励建模和在线策略蒸馏引入了额外约束，例如依赖专门的奖励模型，或假设教师与学生具有相同的推理模式。然而，我们观察到，一旦推理过程首次出现错误，对其后续推理的评估所能提供的额外信息十分有限，因为此时评估已基于一个无效前缀进行条件化。因此，我们提出Cliff，一种奖励塑形策略，利用现成的大语言模型作为教师来识别每次生成轨迹中的首个错误。由此，轨迹被自然地分解为两个部分：正确前缀和错误后缀。Cliff将该信号转化为词元级优势，对正确前缀赋予正向优势，并对其后的部分施加负向反馈。在12种不同场景下的实验表明，Cliff能够持续提升推理性能，相较于在线策略蒸馏平均提升15%，相较于标准GRPO平均提升7%，即使在教师模型能力有限的情况下亦然。此外，我们分析了“真值”在Cliff中的作用，并研究了其训练动态。这些结果确立了Cliff作为一种简单、通用且有效的方法，通过更丰富的细粒度监督来改进RLVR。

English

Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for large language model (LLM) post-training, but its reliance on coarse outcome rewards leads to limited guidance on intermediate reasoning processes. Existing approaches such as process reward modeling and on-policy distillation introduce additional constraints, such as reliance on a specialized reward model or assuming identical reasoning patterns between teacher and student. Nevertheless, we observe that once a reasoning process first goes wrong, evaluating the subsequent reasoning provides limited additional information, as it is already conditioned on an invalid prefix. Therefore, we propose Cliff, a reward shaping strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout. As a result, the rollout is naturally decomposed into two parts: a correct prefix and an incorrect suffix. Cliff then converts this signal into token-level advantages, assigning positive advantages for the correct prefix and negative feedback afterward. Experiments across 12 different scenarios demonstrate that Cliff consistently improves reasoning performance, outperforming on-policy distillation by 15% and standard GRPO by 7%, even with teachers of modest capability. Furthermore, we analyse the role of ``ground truth'' in Cliff and investigate its training dynamics. These results establish Cliff as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision.