Muon何时对智能体强化学习有帮助？

摘要

Muon在大型预训练中与AdamW不相上下，但其在强化学习（RL）后训练中的价值尚不明确。我们通过匹配单种子的对照实验，使用Qwen2.5-0.5B-Instruct在ALFWorld上比较了朴素Muon与AdamW在稀疏奖励的智能体强化学习中的表现。在组内组策略优化（GiGPO）下，仅对隐藏权重矩阵应用Muon使最终窗口的验证成功率从0.290提升至0.546（提升88%）；高学习率的AdamW对照组在更新后未取得成功率提升。该效果取决于优势估计子和学习率。在3e-5学习率下，Muon将GRPO成功率从0.161提升至0.268，而GraphGPO的后期窗口差距在接近饱和时缩小。在1e-5学习率下，GraphGPO Muon达到0.901的成功率，将归一化验证AUC从0.399提升至0.556，并且分别提前30次和60次更新达到0.5和0.75的成功率。这些探索性结果表明，Muon能够有益于智能体强化学习，并激励我们联合研究策略优化器、优势估计子和学习率。多种子及跨任务验证仍有待深入探究。

English

Muon is competitive with AdamW in large-scale pre-training, but its value for reinforcement-learning (RL) post-training remains unclear. We study vanilla Muon in sparse-reward agentic RL through matched single-seed comparisons with AdamW on ALFWorld using Qwen2.5-0.5B-Instruct. Under Group-in-Group Policy Optimization (GiGPO), applying Muon only to hidden weight matrices raises final-window validation success from 0.290 to 0.546 (+88%); high-rate AdamW controls retain no post-update success. The effect depends on the advantage estimator and learning rate. At 3e-5, Muon improves GRPO from 0.161 to 0.268, whereas GraphGPO's late-window gap narrows near saturation. At 1e-5, GraphGPO Muon reaches 0.901, raises normalized validation AUC from 0.399 to 0.556, and reaches 0.5 and 0.75 success 30 and 60 updates earlier, respectively. These exploratory results show that Muon can benefit agentic RL and motivate studying the policy optimizer, advantage estimator, and learning rate jointly. Multi-seed and cross-task validation remain open.