學習尚未掌握的，而非已精通的：多獎勵策略最佳化中的飽和感知優勢重加權

摘要

具有群體相對優勢的強化學習（RL）已成為後訓練語言模型推理模型的事實標準。然而，在優化多個獎勵目標時，現有方法通常先在群體標準化之前，以固定加權和對獎勵向量進行標量化。我們表明，這種設計會導致兩個根本問題：具有不同獎勵特徵的軌跡可能獲得相同的優勢值；而且所有目標都以固定相對權重進行優化，無論其當前飽和程度如何。因此，訓練會持續將梯度預算分配給已解決的目標，而非專注於剩餘空間更大的目標。我們提出了用於多獎勵策略優化的飽和感知優勢重加權（SA-MRPO），該方法獨立地標準化每個獎勵目標，並根據目標飽和度的批次級估計，自適應地折扣其貢獻。這會動態地將優化努力重新分配給優化不足的目標，同時在實證上維持那些已充分滿足目標的性能。我們進一步表明，飽和感知重加權可以翻轉更新的符號，而不僅僅是重新縮放其幅度。在具有雙目標和三目標獎勵組合的數學推理中，SA-MRPO在15項基準比較中的12項中，於更困難的正確性目標上優於GDPO，在AIME24上的提升高達5%。在適應性推理方面，它在全部五個基準上提高了準確率，平均提高3.8%，在AMC23上最高提高9.2%；在程式碼基準上，通過率最高提高2.3%；同時在所有設定中，較容易的目標保持在其已滿足的水準附近。

English

Reinforcement learning (RL) with group-relative advantages has become the de facto standard for post-training language model reasoners. However, when optimizing multiple reward objectives, existing methods typically scalarize the reward vector with a fixed weighted sum before group-wise standardization. We show that this design leads to two fundamental problems: rollouts with distinct reward profiles can receive identical advantages, and all objectives are optimized with fixed relative weights regardless of their current level of saturation. As a result, training continues to allocate gradient budget to already-solved objectives instead of focusing on those with greater remaining headroom. We introduce Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization (SA-MRPO), which standardizes each reward objective independently and adaptively discounts its contribution according to a batch-level estimate of objective saturation. This dynamically reallocates optimization effort toward under-optimized objectives while empirically maintaining performance on those that are already well satisfied. We further show that saturation-aware reweighting can reverse the sign of an update, rather than merely rescale its magnitude. Across mathematical reasoning with two- and three-objective reward combinations, SA-MRPO improves the harder correctness objective over GDPO in 12 of 15 benchmark comparisons, with gains of up to 5% on AIME24. On adaptive reasoning it improves accuracy on all five benchmarks, by 3.8% on average and up to 9.2 % on AMC23, and on coding benchmarks it improves pass rate by up to 2.3%, while in all settings maintaining the easier objectives near their already satisfied levels.