超越欧几里得裁剪：通过黎曼等距策略优化克服LLM强化学习中的探索崩溃

摘要

强化学习（RL）已成为增强大语言模型（LLM）推理能力的主导范式。然而，基于PPO-Clip的强化学习算法本质上受限于探索崩溃。后续工作仍主要停留在启发式层面，未能识别PPO-Clip失效的根本原因。本文揭示了PPO-Clip的根本缺陷：其隐式地使用欧几里得度量来衡量策略差异，这在理论上与策略黎曼流形上的固有几何结构不一致。这种几何不匹配导致在低概率区域更新过于保守，而在高概率区域更新过于激进，最终导致探索崩溃。为纠正这一几何缺陷，我们提出黎曼等距策略优化（RIPO），该方法保证在黎曼流形上进行等距策略更新，有效平衡探索与利用。我们进一步证明RIPO实现了有利的偏差-方差权衡，从而稳定优化过程。大量实验表明，RIPO在七个竞赛级基准测试中显著超越现有LLM强化学习算法（在AIME24上相比GRPO提升高达60%）。

English

Reinforcement learning (RL) has become a dominant paradigm for enhancing LLMs' reasoning capabilities. However, RL algorithms with PPO-Clip are inherently limited by exploration collapse. Subsequent works remain primarily heuristic and fail to identify the essential cause of PPO-Clip's failure. This work reveals the fundamental flaw of PPO-Clip: it implicitly measures policy discrepancy using Euclidean metric, which is theoretically inconsistent with the intrinsic geometry on the policy Riemannian manifold. This geometric mismatch results in overly conservative updates in low-probability regions while aggressive in high-probability regions, ultimately collapsing exploration. To correct this geometric flaw, we propose Riemannian Isometric Policy Optimization (RIPO), which guarantees isometric policy updates on the Riemannian manifold, effectively balancing exploration and exploitation. We further show that RIPO achieves a favorable bias-variance trade-off, which stabilizes optimization. Extensive experiments demonstrate that RIPO significantly surpasses existing LLM RL algorithms across seven competition-level benchmarks (up to 60% improvement over GRPO on AIME24).