超越歐幾里得裁剪：透過黎曼等距策略優化克服大型語言模型強化學習中的探索崩潰

摘要

強化學習（RL）已成為提升大型語言模型（LLM）推理能力的主流典範。然而，採用PPO-Clip的強化學習演算法本質上受到探索崩潰的限制。後續研究大多仍停留在啟發式方法，未能辨識出PPO-Clip失效的根本原因。本研究揭示了PPO-Clip的基礎缺陷：它隱含地使用歐幾里得度量來衡量策略差異，這在理論上與策略黎曼流形的內在幾何結構不一致。這種幾何失配導致在低機率區域過度保守更新，而在高機率區域則過於激進，最終使探索崩潰。為矯正此幾何缺陷，我們提出黎曼等距策略優化（RIPO），該方法保證在黎曼流形上進行等距策略更新，有效平衡探索與利用。我們進一步證明RIPO能達到有利的偏差-變異權衡，從而穩定最佳化過程。廣泛的實驗顯示，在七個競賽級基準測試中，RIPO顯著優於現有的LLM強化學習演算法（在AIME24上相較GRPO提升高達60%）。

English

Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs' reasoning capabilities. However, RL algorithms with PPO-Clip are inherently limited by exploration collapse. Subsequent works remain primarily heuristic and fail to identify the essential cause of PPO-Clip's failure. This work reveals the fundamental flaw of PPO-Clip: it implicitly measures policy discrepancy using Euclidean metric, which is theoretically inconsistent with the intrinsic geometry on the policy Riemannian manifold. This geometric mismatch results in overly conservative updates in low-probability regions while aggressive in high-probability regions, ultimately collapsing exploration. To correct this geometric flaw, we propose Riemannian Isometric Policy Optimization (RIPO), which guarantees isometric policy updates on the Riemannian manifold, effectively balancing exploration and exploitation. We further show that RIPO achieves a favorable bias-variance trade-off, which stabilizes optimization. Extensive experiments demonstrate that RIPO significantly surpasses existing LLM RL algorithms across seven competition-level benchmarks (up to 60% improvement over GRPO on AIME24).