RL-PLUS：通过混合策略优化应对大语言模型在强化学习中的能力边界退化问题

RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization

July 31, 2025

作者: Yihong Dong, Xue Jiang, Yongding Tao, Huanyu Liu, Kechi Zhang, Lili Mou, Rongyu Cao, Yingwei Ma, Jue Chen, Binhua Li, Zhi Jin, Fei Huang, Yongbin Li, Ge Li

cs.AI

摘要

基于可验证奖励的强化学习（RLVR）显著提升了大型语言模型（LLMs）的复杂推理能力。然而，由于其本质上采用的在线策略与LLM庞大的动作空间及稀疏奖励相结合，RLVR难以突破基础LLM固有的能力边界。更为关键的是，RLVR可能导致能力边界崩溃，从而缩小LLM的问题解决范围。为解决这一问题，我们提出了RL-PLUS，一种新颖的混合策略优化方法，它通过内部探索与外部数据的协同作用，旨在增强推理能力并超越基础模型的限制。RL-PLUS集成了两大核心组件：多重重要性采样以解决外部数据带来的分布不匹配问题，以及基于探索的优势函数引导模型走向高价值、未探索的推理路径。我们通过理论分析和大量实验验证了该方法的优越性和普适性。与现有RLVR方法相比，RL-PLUS在六个数学推理基准测试中达到了1）最先进的性能；2）在六个分布外推理任务上表现出色；3）在不同模型家族中均实现了持续且显著的提升，平均相对改进高达69.2%。此外，Pass@k曲线分析表明，RL-PLUS有效解决了能力边界崩溃问题。

English

Reinforcement Learning with Verifiable Reward (RLVR) has significantly advanced the complex reasoning abilities of Large Language Models (LLMs). However, it struggles to break through the inherent capability boundaries of the base LLM, due to its essentially on-policy strategy coupled with LLM's immense action space and sparse reward. Critically, RLVR can lead to the capability boundary collapse, narrowing the LLM's problem-solving scope. To address this problem, we propose RL-PLUS, a novel hybrid-policy optimization approach for LLMs that synergizes internal exploitation with external data to achieve stronger reasoning capabilities and surpass the boundaries of base models. RL-PLUS integrates two core components, i.e., Multiple Importance Sampling to address distributional mismatch from external data, and Exploration-Based Advantage Function to guide the model towards high-value, unexplored reasoning paths. We provide both theoretical analysis and extensive experiments to demonstrate the superiority and generalizability of our approach. Compared with existing RLVR methods, RL-PLUS achieves 1) state-of-the-art performance on six math reasoning benchmarks; 2) superior performance on six out-of-distribution reasoning tasks; 3) consistent and significant gains across diverse model families, with average relative improvements up to 69.2\%. Moreover, the analysis of Pass@k curves indicates that RL-PLUS effectively resolves the capability boundary collapse problem.