学习未掌握的，而非已精通的：多奖励策略优化中的饱和感知优势重新加权

摘要

基于组相对优势的强化学习（RL）已成为语言模型推理器后训练的事实标准。然而，在优化多个奖励目标时，现有方法通常在分组标准化之前用固定加权和将奖励向量标量化。我们表明，这种设计导致两个根本性问题：具有不同奖励分布特征的采样轨迹可能获得相同的优势值，且所有目标均以固定的相对权重进行优化，而不考虑其当前的饱和程度。因此，训练持续将梯度预算分配给已经解决的目标，而非聚焦于具有更大剩余优化空间的目标。我们提出面向多奖励策略优化的饱和感知优势重加权方法（SA-MRPO），该方法独立地对每个奖励目标进行标准化，并根据批次级别的目标饱和程度估计自适应地折扣其贡献。这会将优化努力动态重新分配给优化不足的目标，同时在经验上保持已充分满足目标的性能。我们进一步表明，饱和感知重加权能够反转更新的符号，而不仅仅是重新缩放其幅度。在具有双目标和三目标奖励组合的数学推理任务中，SA-MRPO在15项基准比较中的12项上优于GDPO的正确性目标，在AIME24上提升高达5%。在自适应推理任务中，该方法在所有五个基准测试上均提升了准确率，平均提升3.8%，在AMC23上最高提升9.2%；在编程基准测试中，通过率提升最高达2.3%，同时在所有设置中均将较易目标维持在接近其已饱和水平的状态。

English

Reinforcement learning (RL) with group-relative advantages has become the de facto standard for post-training language model reasoners. However, when optimizing multiple reward objectives, existing methods typically scalarize the reward vector with a fixed weighted sum before group-wise standardization. We show that this design leads to two fundamental problems: rollouts with distinct reward profiles can receive identical advantages, and all objectives are optimized with fixed relative weights regardless of their current level of saturation. As a result, training continues to allocate gradient budget to already-solved objectives instead of focusing on those with greater remaining headroom. We introduce Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization (SA-MRPO), which standardizes each reward objective independently and adaptively discounts its contribution according to a batch-level estimate of objective saturation. This dynamically reallocates optimization effort toward under-optimized objectives while empirically maintaining performance on those that are already well satisfied. We further show that saturation-aware reweighting can reverse the sign of an update, rather than merely rescale its magnitude. Across mathematical reasoning with two- and three-objective reward combinations, SA-MRPO improves the harder correctness objective over GDPO in 12 of 15 benchmark comparisons, with gains of up to 5% on AIME24. On adaptive reasoning it improves accuracy on all five benchmarks, by 3.8% on average and up to 9.2 % on AMC23, and on coding benchmarks it improves pass rate by up to 2.3%, while in all settings maintaining the easier objectives near their already satisfied levels.