群組自適應裁剪策略優化

摘要

在可驗證獎勵強化學習（RLVR）中，群體相對策略最佳化（GRPO）通常對所有軌跡使用固定的重要性取樣（IS）比率裁剪邊界。我們指出一個關鍵限制：較難問題上罕見的正確軌跡與較易問題上大量出現的正確軌跡，儘管其學習訊號差異極大，卻以相近的比率被裁剪。群組成功率較低的軌跡具有較大的 IS 比率，並攜帶更強的梯度訊號以促進探索與解決新問題，卻被固定裁剪不成比例地壓抑。為解決此問題，我們提出群體自適應裁剪策略最佳化（GAPO），這是一種可即插即用的 GRPO 修改方式，能根據軌跡優勢調整裁剪邊界。GAPO 的動機來自反向 KL 信賴域觀點，該觀點認為具有較大學習訊號的軌跡應獲得按比例更大的更新空間。GAPO 無需獎勵塑形，且僅調整裁剪閾值，同時保留標準的 PPO/GSPO 代理目標。在 Qwen 與 Llama 模型上，相較於固定裁剪與優勢塑形基線，GAPO 於基礎模型通過率相對較低的數學推理與程式碼基準上，一致地提升 Pass@1 與 Pass@k。

English

Group relative policy optimization for reinforcement learning with verifiable rewards (RLVR) typically uses a fixed importance-sampling (IS) ratio clipping boundary across all rollouts. We identify a key limitation: rare correct rollouts on harder problems and abundant correct rollouts on easier problems are clipped at comparable rates, despite contributing very different learning signals. Rollouts with low group success exhibit larger IS ratios and carry stronger gradient signal for exploration and solving new problems, yet are disproportionately suppressed by fixed clipping. To address this, we propose Group Adaptive Clipping Policy Optimization (GAPO), a plug-in modification to GRPO methods that adapts the clipping boundary to the rollout advantage. GAPO is motivated by a reverse-KL trust-region perspective, which suggests that rollouts with larger learning signal should receive proportionally greater update headroom. GAPO requires no reward shaping and preserves the standard PPO/GSPO surrogate while adapting only the clipping threshold. Across Qwen and Llama models, GAPO consistently improves both Pass@1 and Pass@k over fixed clipping and advantage-shaping baselines on math reasoning and coding benchmarks where the pass rates by the base model are relatively low.