基于可验证奖励的强化学习样本极性重构

摘要

大型推理模型通常采用可验证奖励的强化学习进行训练，以提升其推理能力。在该范式下，策略更新同时利用自我生成的正向与负向推演轨迹，二者对应不同的样本极性。本文系统性地研究了这些样本极性如何影响可验证奖励强化学习的训练动态与行为模式。我们发现正向样本能锐化已有的正确推理模式，而负向样本则能促进对新推理路径的探索。我们进一步探究了在样本级别和标记级别调整正负样本优势值对训练的影响。基于这些发现，我们提出了一种自适应非对称的标记级优势重塑策略优化方法A3PO，该方法能更精准地根据不同极性将优势信号分配至关键标记。在五个推理基准测试上的实验验证了我们方法的有效性。

English

Large reasoning models (LRMs) are typically trained using reinforcement learning with verifiable reward (RLVR) to enhance their reasoning abilities. In this paradigm, policies are updated using both positive and negative self-generated rollouts, which correspond to distinct sample polarities. In this paper, we provide a systematic investigation into how these sample polarities affect RLVR training dynamics and behaviors. We find that positive samples sharpen existing correct reasoning patterns, while negative samples encourage exploration of new reasoning paths. We further explore how adjusting the advantage values of positive and negative samples at both the sample level and the token level affects RLVR training. Based on these insights, we propose an Adaptive and Asymmetric token-level Advantage shaping method for Policy Optimization, namely A3PO, that more precisely allocates advantage signals to key tokens across different polarities. Experiments across five reasoning benchmarks demonstrate the effectiveness of our approach.

基于可验证奖励的强化学习样本极性重构

Rethinking Sample Polarity in Reinforcement Learning with Verifiable Rewards

摘要

Support