Muon 何時有助於代理強化學習？

摘要

Muon在大型預訓練中與AdamW競爭力相當，但其在強化學習（RL）後訓練階段的價值仍不明確。我們透過在ALFWorld環境中使用Qwen2.5-0.5B-Instruct進行單一種子匹配比較，研究原始Muon在稀疏獎勵的智能體強化學習（agentic RL）中的表現。在分組內分組策略優化（GiGPO）下，僅對隱藏權重矩陣應用Muon可將最終視窗驗證成功率從0.290提升至0.546（增幅達88%）；而高學習率的AdamW控制組則未出現更新後的成功。此效果取決於優勢估計器與學習率。在學習率3e-5時，Muon將GRPO從0.161提升至0.268，而GraphGPO的晚期視窗差距則接近飽和。在學習率1e-5時，GraphGPO搭配Muon可達到0.901，並將標準化驗證AUC從0.399提升至0.556，且分別提前30次與60次更新即達到0.5與0.75的成功率。這些探索性結果顯示Muon能為智能體強化學習帶來益處，並激勵我們聯合研究策略優化器、優勢估計器與學習率。多種子與跨任務驗證仍為待解決的課題。

English

Muon is competitive with AdamW in large-scale pre-training, but its value for reinforcement-learning (RL) post-training remains unclear. We study vanilla Muon in sparse-reward agentic RL through matched single-seed comparisons with AdamW on ALFWorld using Qwen2.5-0.5B-Instruct. Under Group-in-Group Policy Optimization (GiGPO), applying Muon only to hidden weight matrices raises final-window validation success from 0.290 to 0.546 (+88%); high-rate AdamW controls retain no post-update success. The effect depends on the advantage estimator and learning rate. At 3e-5, Muon improves GRPO from 0.161 to 0.268, whereas GraphGPO's late-window gap narrows near saturation. At 1e-5, GraphGPO Muon reaches 0.901, raises normalized validation AUC from 0.399 to 0.556, and reaches 0.5 and 0.75 success 30 and 60 updates earlier, respectively. These exploratory results show that Muon can benefit agentic RL and motivate studying the policy optimizer, advantage estimator, and learning rate jointly. Multi-seed and cross-task validation remain open.