超越正确性：通过强化学习训练实现过程与结果奖励的和谐统一

Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training

September 3, 2025

作者: Chenlu Ye, Zhou Yu, Ziji Zhang, Hao Chen, Narayanan Sadagopan, Jing Huang, Tong Zhang, Anurag Beniwal

cs.AI

摘要

可驗證獎勵的強化學習（RLVR）已成為數學推理任務的主流範式，在推理能力上提供了穩定的提升。然而，RLVR中的結果獎勵模型（ORMs）過於粗粒度，無法區分正確答案中的錯誤推理或錯誤答案中的有效推理。這種缺乏細粒度的情況顯著引入了噪聲和誤導性的梯度，阻礙了推理過程質量的進一步提升。雖然過程獎勵模型（PRMs）為中間步驟提供了細粒度的指導，但它們經常存在不準確性，並且容易受到獎勵欺騙的影響。 為了解決這一困境，我們引入了過程一致性過濾器（PROF），這是一種有效的數據處理策展方法，它將噪聲的細粒度過程獎勵與準確的粗粒度結果獎勵相協調。與在目標函數中簡單混合PRM和ORM（arXiv:archive/2506.18896）不同，PROF通過一致性驅動的樣本選擇來利用它們的互補優勢。我們的方法保留了具有較高平均過程值的正確響應和具有較低平均過程值的錯誤響應，同時保持了正/負訓練樣本的平衡。大量實驗表明，我們的方法不僅在最終準確性上比混合方法持續提高了超過4%，而且還增強了中間推理步驟的質量。代碼和訓練配方可在https://github.com/Chenluye99/PROF獲取。

English

Reinforcement learning with verifiable rewards (RLVR) has emerged to be a predominant paradigm for mathematical reasoning tasks, offering stable improvements in reasoning ability. However, Outcome Reward Models (ORMs) in RLVR are too coarse-grained to distinguish flawed reasoning within correct answers or valid reasoning within incorrect answers. This lack of granularity introduces noisy and misleading gradients significantly and hinders further progress in reasoning process quality. While Process Reward Models (PRMs) offer fine-grained guidance for intermediate steps, they frequently suffer from inaccuracies and are susceptible to reward hacking. To resolve this dilemma, we introduce PRocess cOnsistency Filter (PROF), an effective data process curation method that harmonizes noisy, fine-grained process rewards with accurate, coarse-grained outcome rewards. Rather than naively blending PRM and ORM in the objective function (arXiv:archive/2506.18896), PROF leverages their complementary strengths through consistency-driven sample selection. Our approach retains correct responses with higher averaged process values and incorrect responses with lower averaged process values, while maintaining positive/negative training sample balance. Extensive experiments demonstrate that our method not only consistently improves the final accuracy over 4% compared to the blending approaches, but also strengthens the quality of intermediate reasoning steps. Codes and training recipes are available at https://github.com/Chenluye99/PROF.