UP：用於打破探索-穩定性困境的無界正非對稱優化

摘要

強化學習（RL）已成為提升大型語言模型（LLMs）複雜推理能力的標準框架。為實現樣本效率，現代 RL 框架依賴於重要性採樣（IS）。然而，這類演算法面臨探索與穩定性的兩難困境：純粹的 IS 常導致災難性的訓練不穩定，而為緩解此問題所使用的標準裁剪機制則嚴格限制了策略更新預算。透過形式化「概率容量」（Cap）概念，我們揭示出保守的裁剪機制會過早截斷正確但低置信度推理路徑的更新預算，從而在結構上抑制探索。為突破這些限制，我們提出「無界正向非對稱優化」（UP），這是一種通用且即插即用的目標函數。UP 透過停止梯度運算元將策略錨定於當前狀態，從理論上重構優化過程。此非對稱設計能為正向優勢提供未經裁剪的穩定梯度，以最大化探索能力；同時對負向優勢保留標準裁剪機制，以防止訓練不穩定。此外，我們的公式可輕鬆擴展至不同優化粒度，包括詞元層級（GRPO、DAPO）與序列層級（GSPO）框架。大量實驗證明，UP 能增強探索能力，並在多元 RL 演算法（DAPO、GSPO 及 GRPO）、模型架構（密集、混合專家及視覺語言）以及訓練模式（語言與多模態）中獲得優越的推理準確率，驗證了 UP 作為基於 RL 訓練的通用即插即用增強工具之有效性。

English

Reinforcement learning (RL) has become the standard paradigm for enhancing the complex reasoning capabilities of large language models (LLMs). To achieve sample efficiency, modern RL frameworks rely on importance sampling (IS). However, these algorithms suffer from an exploration-stability dilemma. Pure IS often leads to catastrophic training instability, while standard clipping mechanisms used to mitigate this instability strictly constrain the policy update budget. By formalizing the concept of Probability Capacity (Cap), we reveal that conservative clipping structurally stifles exploration by prematurely truncating the update budget for correct but low-confidence reasoning paths. To break free from these constraints, we propose Unbounded Positive Asymmetric Optimization (UP), a universal and plug-and-play objective. UP theoretically restructures the optimization process by anchoring the policy to its current state via the stop-gradient operator. This asymmetric design unleashes unclipped, stable gradients for positive advantages to maximize exploration, while maintaining standard clipping safeguards for negative advantages to prevent training instability. Furthermore, our formulation readily extends across different optimization granularities, including token-level (GRPO, DAPO) and sequence-level (GSPO) frameworks. Extensive experiments demonstrate that UP enhances exploration capacity and achieves superior reasoning accuracy across diverse RL algorithms (DAPO, GSPO, and GRPO), model architectures (Dense, MoE, and vision-language), and training modalities (language and multimodal), validating UP as a truly universal plug-and-play enhancement for RL-based training.