智能体强化学习的单次展开异步优化

摘要

强化学习（RL）在大型语言模型（LLM）的后训练阶段正变得日益重要。以往的LLM强化学习流程多为同步且批次交错的模式，这在面向长期智能体任务时效率低下。近期，异步强化学习作为一种更高效的替代方案崭露头角，它通过随着轨迹样本到达而实时更新模型。然而，现有异步强化学习系统往往侧重吞吐量，而对训练稳定性及任务有效性的探索相对不足。例如，一个关键挑战在于广泛使用的GRPO框架中的分组采样机制天然不适用于异步智能体训练。本文提出单轨迹异步优化（SAO）方法，以应对异步强化学习中的稳定性和离策略挑战。为减少离策略效应并提升泛化能力，我们用单轨迹采样替代分组采样，即每个提示仅对应一次轨迹采样。我们进一步通过实用的价值模型训练设计改进这一单轨迹策略。为提升优化稳定性，我们引入严格的双侧令牌级裁剪策略。SAO能够稳定训练上千步，并在智能体编程与推理基准测试（如SWE-Bench Verified、BeyondAIME和IMOAnswerBench）上持续优于GRPO及其变体。我们还证明，在模拟在线学习场景中（模型需适应动态变化的环境），单轨迹强化学习尤为有效。基于此，SAO已成功部署于开源GLM-5.2模型（750B-A40B）的智能体强化学习训练流程中。

English

Reinforcement learning (RL) is becoming increasingly important for post-training large language models (LLMs). Previous RL pipelines for LLMs were mostly synchronous and batch-interleaved, which is inefficient for long-horizon agentic tasks. Recently, asynchronous RL has emerged as a more efficient alternative by updating the model as rollouts arrive. However, existing asynchronous RL systems often emphasize throughput, while leaving training stability and task effectiveness largely underexplored. For example, a key challenge is that group-wise sampling in the widely-used GRPO framework does not naturally fit asynchronous agentic training. In this paper, we present Single-rollout Asynchronous Optimization (SAO) to address the stability and off-policy challenges in asynchronous RL. To reduce off-policy effects and improve generalization, we replace group-wise sampling with single-rollout sampling, that is, using one rollout per prompt. We further improve this single-rollout strategy with practical value-model training designs. To improve optimization stability, we introduce a strict double-side token-level clipping strategy. SAO is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks, such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. We also demonstrate that single-rollout RL is particularly effective in a simulated online learning setting, where the model must adapt to changing evolving environments. To this end, SAO is successfully deployed in the agentic RL pipeline for training the open GLM-5.2 model (750B-A40B).