超越稳定性-探索困境：面向大语言模型策略优化的环境正则化方法

摘要

大语言模型（LLM）的策略优化（Policy Optimization, PO）面临稳定性与探索性之间的权衡，目前通过动作侧的策略KL正则化器（Policy-KL regularizer）来调节。这使实践者陷入两难困境：保留Policy-KL会约束响应行为并消耗动作侧的探索预算，而放弃它则使优化过程缺乏显式的漂移控制。我们提出一种替代方案，通过将正则化移至输入侧来打破这一困境。随着训练的进行，当前策略所导致的训练查询分布会不受约束地偏离其RL前的参考分布。具体来说，环境正则化策略优化（ERPO）引入了一个查询KL（Query-KL, QKL）项来约束这种查询分布偏移，同时引入一个基于数据集静态参考的逐查询权重，该权重使每次查询更新偏向于参考分布下典型的查询。QKL梯度严格通过查询似然传播；策略梯度估计器所使用的响应得分函数不出现在QKL项中，因此QKL不会对响应分布产生直接的梯度压力——从而保留了探索性。ERPO可无缝接入GRPO/PPO/REINFORCE风格的流程，且无需额外的前向传播。在六个数学推理基准上，ERPO替代了标准Policy-KL正则化器，同时实现了对查询分布漂移的有效控制，在高温解码和长时程训练下提供了更强的准确性和显著更稳定的行为。我们的源代码可在 https://github.com/alibaba/ERPO 获取。

English

Policy optimization (PO) for Large Language Models faces a stability--exploration trade-off, currently mediated by an action-side Policy-KL regularizer. This puts practitioners in a double bind: keeping Policy-KL constrains response behavior and consumes the action-side exploration budget, while dropping it leaves the optimization without an explicit drift control. We argue for an alternative that breaks the dilemma by moving regularization to the input side. As training progresses, the distribution over training queries induced by the current policy drifts unchecked from its pre-RL reference distribution. Concretely, Environment-Regularized Policy Optimization (ERPO) introduces a Query-KL (QKL) term that bounds this query distribution shift, together with a dataset-static reference-derived per-query weight that biases each per-query update toward queries typical under the reference. The QKL gradient flows strictly through the query likelihood; the response score function used by policy-gradient estimators does not appear in the QKL term, so QKL exerts no direct gradient pressure on the response distribution---exploration is preserved. ERPO plugs into GRPO/PPO/REINFORCE-style pipelines without additional forward passes. On six mathematical reasoning benchmarks, ERPO replaces the standard Policy-KL regularizer while achieving effective control over query distribution drift, delivering stronger accuracy and substantially more stable behavior under high-temperature decoding and long-horizon training.Our source code are available at https://github.com/alibaba/ERPO