自主強化學習之單次展開非同步優化

摘要

強化學習在大型語言模型後訓練中的應用日益重要。先前的LLM強化學習流程多採用同步與批次交錯模式，在處理長時域智能體任務時效率低下。近期，非同步強化學習透過在軌跡生成時即時更新模型，成為更高效的替代方案。然而，現有非同步強化學習系統往往著重於吞吐量，對訓練穩定性與任務有效性探討不足。例如，廣泛使用的GRPO框架中的分組採樣機制難以自然契合非同步智能體訓練。為解決非同步強化學習中的穩定性與異策略挑戰，本文提出單一軌跡非同步優化方法。為降低異策略效應並提升泛化能力，我們以單一軌跡採樣取代分組採樣，即每個提示僅使用一條軌跡。我們進一步結合實用的價值模型訓練設計優化了單軌跡策略。為提升優化穩定性，我們引入了嚴格的雙側詞元級裁剪策略。SAO能夠穩定訓練超過一千步，並在智慧體編碼與推理基準（如SWE-Bench Verified、BeyondAIME及IMOAnswerBench）上持續優於GRPO及其變體。我們同時證明，在模擬線上學習環境中（模型需適應動態變化的環境），單軌跡強化學習特別有效。為此，SAO已成功部署於開源GLM-5.2模型（750B-A40B）的智能體強化學習訓練流程中。

English

Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).