LongStraw: 在固定GPU预算下实现超过200万token的长上下文强化学习

LongStraw: Long-Context RL Beyond 2M Tokens under a Fixed GPU Budget

July 16, 2026

作者: Changhai Zhou, Kieran Liu, Yuhua Zhou, Qian Qiao, Jun Gao, Harry Zhang, Irvine Lu, Nolan Ho, Lucian Li, Andrew Lei, Cleon Cheng, Steven Chiang, Yihang Zeng, Di Zhang, Rio Yang, Kaijie Chen, Andrew Chen, Pony Ma, Weizhong Zhang, Cheng Jin

cs.AI

摘要

推理上下文长度与强化学习后训练之间的差距正在不断扩大：推理系统已接近百万token级的上下文，而后训练任务通常仍保持在256K token或以下，并依赖部署时的长度泛化。这一差距对AI智能体尤为重要——其观测结果、工具输出、文档以及先前决策会在长时间轨迹中不断累积。LongStraw是一种在固定GPU预算下实现百万token级强化学习后训练的架构感知执行栈，基于分组相对策略优化（GRPO）实现。它评估共享提示时无需自动求导，仅保留后续token所需的模型特定状态，并逐一重放短响应分支，以增加重放时间为代价缩减实时训练图。我们针对混合循环与全注意力机制的Qwen3.6-27B模型以及压缩注意力混合专家模型GLM-5.2实现了该方案。在八块H20 GPU上，LongStraw在分组大小为2和8时完成了2.1M位置的分组Qwen评分与响应反向传播；将分组大小增加仅需额外0.21 GB峰值内存，而单独的压力测试则达到了4.46M位置。在32块H20 GPU上，我们验证了GLM-5.2全部78层在2.1M token提示下的端到端LongStraw执行路径。这些实验确立了执行能力而非完整的训练正确性，因为捕获的提示状态是分离的，且部分分布式前向传播与梯度组合路径尚未完善。

English

A growing gap separates inference context lengths from RL post-training: inference systems are approaching million-token contexts, while post-training workloads often remain at 256K tokens or below and rely on length generalization at deployment. The gap is especially important for AI agents, whose observations, tool outputs, documents, and prior decisions accumulate over long trajectories. LongStraw is an architecture-aware execution stack for million-token RL post-training under a fixed GPU budget, instantiated with Group Relative Policy Optimization (GRPO). It evaluates the shared prompt without autograd, retains only model-specific state needed by later tokens, and replays short response branches one at a time, reducing the live training graph at the cost of additional replay time. We implement it for the hybrid recurrent and full-attention Qwen3.6-27B and the compressed-attention mixture-of-experts GLM-5.2. On eight H20 GPUs, LongStraw completes grouped Qwen scoring and response backward at 2.1M positions for groups of 2 and 8; increasing the group size adds only 0.21 GB of peak allocated memory, while a separate stress test reaches 4.46M positions. On 32 H20 GPUs, we validate the end-to-end LongStraw execution path for a 2.1M-token prompt across all 78 layers of GLM-5.2. These experiments establish execution capacity rather than complete training correctness because the captured prompt state is detached and some distributed forward and gradient composition paths remain incomplete.