LEGO-RL：面向編碼代理的框架原生強化學習

摘要

針對編碼智能體的強化學習日益依賴長時間運行的智能體執行框架，以管理工具整合、程式碼庫上下文與執行反饋。然而，這些執行框架的原生執行環境本質上與策略梯度訓練不一致：環境崩潰與獎勵作弊會破壞結果信號，而訓練-推理差異則使採樣行為與策略更新脫節。為解決此問題，我們提出 LEGO-RL，這是一個在不修改原生編碼智能體執行框架內部控制流程的前提下，將其與可擴展的策略梯度優化橋接起來的框架。LEGO-RL 建立在三大支柱之上：(1) 透過進程內 LLM 代理實現忠實優化，該機制捕捉原始生成串流以進行 Token 層級的對齊，並在框架端進行壓縮或重新序列化時，仍能穩健地在訓練端重新計算對數機率；(2) 透過具備映像快取與分階段防禦的可擴展沙盒編排實現可靠執行，以緩解獎勵作弊；(3) 透過整合式外掛模組實現可觀測訓練，該模組自動化驗證與監控，並搭配即時 UI 以進行細粒度軌跡診斷。我們在三種原生編碼智能體執行框架上，使用 GSPO 訓練稀疏 MoE 模型 Qwen3.5-35B-A3B，以此評估 LEGO-RL。在 SWE-bench Verified 上，LEGO-RL 使 Qwen3.5-35B-A3B 於 OpenHands SDK（64.0% 提升至 70.4%）、Claude Code（62.4% 提升至 68.2%）及 OpenCode（57.2% 提升至 66.6%）上的表現均獲得改善，同時維持高於 0.99 的採樣-訓練機率相關性。

English

Reinforcement learning for coding agents increasingly relies on long-running agent harnesses to manage tool integration, repository contexts, and execution feedback. However, the native execution environments of these harnesses are inherently misaligned with policy-gradient training: environmental crashes and reward hacking corrupt outcome signals, while train-inference discrepancies decouple rollout behavior from policy updates. To address this, we present LEGO-RL, a framework that bridges native coding-agent harnesses with scalable policy-gradient optimization without modifying their internal control flow. LEGO-RL is built upon three pillars: (1) faithful optimization via in-process LLM proxying that captures raw generation streams for token-level alignment and robust trainer-side log-probability recomputation, even under harness-side compaction or re-serialization; (2) reliable execution via scalable sandbox orchestration featuring image caching and stage-wise defenses to mitigate reward hacking; and (3) observable training through an integrated plugin that automates validation and monitoring, paired with a Live UI for granular trajectory diagnostics. We evaluate LEGO-RL by training the sparse MoE model Qwen3.5-35B-A3B with GSPO across three native coding-agent harnesses. LEGO-RL improves Qwen3.5-35B-A3B across OpenHands SDK (64.0% to 70.4%), Claude Code (62.4% to 68.2%), and OpenCode (57.2% to 66.6%) on SWE-bench Verified, while maintaining a rollout-training probability correlation above 0.99.