超越穩定-探索困境：大型語言模型策略優化中的環境正則化

摘要

針對大型語言模型的策略優化（PO）面臨穩定性與探索之間的權衡，目前透過動作側的策略KL正則化器來調節。這使從業者陷入雙重束縛：保留策略KL會約束回應行為並消耗動作側的探索預算；而將其移除則使優化過程缺乏明確的漂移控制。我們提出另一種替代方案，透過將正則化移至輸入側來打破此困境。隨著訓練進行，當前策略所誘導的訓練查詢分佈會不受控制地偏離其在強化學習前的參考分佈。具體而言，環境正則化策略優化（ERPO）引入了查詢KL（QKL）項，用以限制查詢分佈偏移，並結合由資料集靜態參考推導出的逐查詢權重，使每次逐查詢更新偏向於參考下的典型查詢。QKL梯度嚴格地經由查詢似然傳遞；策略梯度估計器所使用的回應評分函數不會出現在QKL項中，因此QKL對回應分佈不施加直接的梯度壓力——探索因而得以保留。ERPO可整合至GRPO/PPO/REINFORCE風格的流程中，且無需額外的前向傳播。在六個數學推理基準上，ERPO取代了標準的策略KL正則化器，同時實現對查詢分佈漂移的有效控制，並帶來更強的準確性，且在高溫解碼與長視野訓練下表現顯著更穩定。我們的原始碼可在 https://github.com/alibaba/ERPO 取得。

English

Policy optimization (PO) for Large Language Models faces a stability--exploration trade-off, currently mediated by an action-side Policy-KL regularizer. This puts practitioners in a double bind: keeping Policy-KL constrains response behavior and consumes the action-side exploration budget, while dropping it leaves the optimization without an explicit drift control. We argue for an alternative that breaks the dilemma by moving regularization to the input side. As training progresses, the distribution over training queries induced by the current policy drifts unchecked from its pre-RL reference distribution. Concretely, Environment-Regularized Policy Optimization (ERPO) introduces a Query-KL (QKL) term that bounds this query distribution shift, together with a dataset-static reference-derived per-query weight that biases each per-query update toward queries typical under the reference. The QKL gradient flows strictly through the query likelihood; the response score function used by policy-gradient estimators does not appear in the QKL term, so QKL exerts no direct gradient pressure on the response distribution---exploration is preserved. ERPO plugs into GRPO/PPO/REINFORCE-style pipelines without additional forward passes. On six mathematical reasoning benchmarks, ERPO replaces the standard Policy-KL regularizer while achieving effective control over query distribution drift, delivering stronger accuracy and substantially more stable behavior under high-temperature decoding and long-horizon training.Our source code are available at https://github.com/alibaba/ERPO