LEGO-RL：面向代码智能体的测试框架原生强化学习

摘要

面向代码智能体的强化学习日益依赖长时间运行的智能体框架，以管理工具集成、代码仓库上下文和执行反馈。然而，这些框架的原生执行环境与策略梯度训练天然存在不一致性：环境崩溃和奖励黑客行为会破坏结果信号，而训练-推理差异则使轨迹生成行为与策略更新解耦。为解决这一问题，我们提出了 LEGO-RL，一个将原生代码智能体框架与可扩展的策略梯度优化相连接且无需修改其内部控制流的框架。LEGO-RL 建立在三大支柱之上：（1）保真优化，通过进程内 LLM 代理捕获原始生成流，实现词元级对齐和稳健的训练端对数概率重算，即使在框架端进行压缩或重新序列化的情况下依然如此；（2）可靠执行，通过可扩展的沙箱编排，结合镜像缓存与分阶段防御，以缓解奖励黑客行为；（3）可观测训练，借助集成插件自动完成验证与监控，并配合 Live UI 提供细粒度轨迹诊断。我们使用 GSPO 在三个原生代码智能体框架上训练稀疏 MoE 模型 Qwen3.5-35B-A3B，以此评估 LEGO-RL。在 SWE-bench Verified 上，LEGO-RL 将 Qwen3.5-35B-A3B 在 OpenHands SDK（64.0% 提升至 70.4%）、Claude Code（62.4% 提升至 68.2%）和 OpenCode（57.2% 提升至 66.6%）上的表现均予以提升，同时保持轨迹生成-训练概率相关性高于 0.99。

English

Reinforcement learning for coding agents increasingly relies on long-running agent harnesses to manage tool integration, repository contexts, and execution feedback. However, the native execution environments of these harnesses are inherently misaligned with policy-gradient training: environmental crashes and reward hacking corrupt outcome signals, while train-inference discrepancies decouple rollout behavior from policy updates. To address this, we present LEGO-RL, a framework that bridges native coding-agent harnesses with scalable policy-gradient optimization without modifying their internal control flow. LEGO-RL is built upon three pillars: (1) faithful optimization via in-process LLM proxying that captures raw generation streams for token-level alignment and robust trainer-side log-probability recomputation, even under harness-side compaction or re-serialization; (2) reliable execution via scalable sandbox orchestration featuring image caching and stage-wise defenses to mitigate reward hacking; and (3) observable training through an integrated plugin that automates validation and monitoring, paired with a Live UI for granular trajectory diagnostics. We evaluate LEGO-RL by training the sparse MoE model Qwen3.5-35B-A3B with GSPO across three native coding-agent harnesses. LEGO-RL improves Qwen3.5-35B-A3B across OpenHands SDK (64.0% to 70.4%), Claude Code (62.4% to 68.2%), and OpenCode (57.2% to 66.6%) on SWE-bench Verified, while maintaining a rollout-training probability correlation above 0.99.