Cliff：從首次錯誤中學習過程獎勵

摘要

可驗證獎勵強化學習（RLVR）已成為大型語言模型（LLM）後訓練的有力典範，但其依賴粗粒度的結果獎勵，導致對中間推理過程的引導有限。現有方法如過程獎勵建模與同策略蒸餾，會引入額外限制，例如依賴專門的獎勵模型，或假設教師與學生的推理模式相同。然而，我們觀察到，一旦推理過程先出現錯誤，評估後續推理所能提供的額外資訊有限，因為評估已以無效前綴為條件。因此，我們提出 Cliff，一種獎勵塑形策略，利用現成的 LLM 作為教師，識別每次軌跡中的第一個錯誤。如此一來，軌跡自然被分解為兩部分：正確的前綴與錯誤的後綴。Cliff 接著將此訊號轉換為 token 級別的優勢，對正確前綴賦予正優勢，並在之後給予負向回饋。在 12 種不同情境下的實驗顯示，Cliff 能一致地提升推理表現，即使使用能力一般的教師，仍比同策略蒸餾高 15%、比標準 GRPO 高 7%。此外，我們分析了「真實答案（ground truth）」在 Cliff 中的作用，並探討其訓練動態。這些結果確立 Cliff 為一種簡單、通用且有效的方法，能透過更豐富、更細粒度的監督來改善 RLVR。

English

Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for large language model (LLM) post-training, but its reliance on coarse outcome rewards leads to limited guidance on intermediate reasoning processes. Existing approaches such as process reward modeling and on-policy distillation introduce additional constraints, such as reliance on a specialized reward model or assuming identical reasoning patterns between teacher and student. Nevertheless, we observe that once a reasoning process first goes wrong, evaluating the subsequent reasoning provides limited additional information, as it is already conditioned on an invalid prefix. Therefore, we propose Cliff, a reward shaping strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout. As a result, the rollout is naturally decomposed into two parts: a correct prefix and an incorrect suffix. Cliff then converts this signal into token-level advantages, assigning positive advantages for the correct prefix and negative feedback afterward. Experiments across 12 different scenarios demonstrate that Cliff consistently improves reasoning performance, outperforming on-policy distillation by 15% and standard GRPO by 7%, even with teachers of modest capability. Furthermore, we analyse the role of ``ground truth'' in Cliff and investigate its training dynamics. These results establish Cliff as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision.