Agent Lightning v1.0：邁向受控智能體強化學習

摘要

現代智能體在智能體外殼（agent harness）中運行，外殼負責管理工具、上下文與控制流，因此外殼成為智能體系統中的關鍵組成部分。我們最初的 Agent Lightning 提出了一種分離式架構，透過 LLM 端點代理器將任意智能體與 RL 訓練連接起來；此方法後來被 verl Uni-Agent、AReaL 2.0、slime 及 Polar 等框架採用。我們將此範式稱為外殼式智能體強化學習（harnessed agentic RL），其中部署時期的外殼直接參與模型後訓練。外殼式智能體強化學習與傳統智能體強化學習存在根本差異：環境互動循環由外殼掌控，而非訓練引擎；訓練器僅能觀察到 LLM 請求-回應對的序列。這在重新分詞、樣本合併、優勢計算、損失正規化與後端排程等方面引入了挑戰，而這些挑戰可能對訓練穩定性與效果產生顯著影響。我們提出了 Agent Lightning v1.0，一個以約 3,500 行程式碼實作的輕量級外殼式智能體強化學習框架。它支援任意智能體外殼，並可作為研究上述挑戰的實用測試平台。我們在指令跟隨、搜尋與程式碼生成智能體上對其進行評估，並提供一套完整的可重現管線，用於程式碼智能體強化學習。僅使用 6,000 筆訓練樣本與適度的運算資源，RL 在 SWE-bench Verified 上將 Qwen3.5-9B 的表現從 41.8% 提升至 56.4%，絕對增幅達 14.6 個百分點。我們釋出完整的工作流程與訓練腳本，以促進外殼式智能體強化學習的可重現研究。

English

Modern agents operate inside agent harnesses that manage tools, context, and control flow, making the harness a critical part of the agent system. Our original Agent Lightning introduced a disaggregated architecture that connects arbitrary agents to RL training through an LLM endpoint proxy, an approach later adopted by frameworks such as verl Uni-Agent, AReaL 2.0, slime, and Polar. We refer to this paradigm as harnessed agentic RL, where the deploy-time harness directly participates in model post-training. Harnessed agentic RL differs fundamentally from traditional agentic RL: the harness, rather than the training engine, owns the environment interaction loop, while the trainer observes only sequences of LLM request-response pairs. This introduces challenges in retokenization, sample merging, advantage calculation, loss normalization, and backend scheduling, which can substantially affect training stability and effectiveness. We present Agent Lightning v1.0, a lightweight framework for harnessed agentic RL implemented in approximately 3,500 lines of code. It supports arbitrary agent harnesses and serves as a practical testbed for studying these challenges. We evaluate it on instruction-following, search, and coding agents, and provide a complete reproducible pipeline for coding-agent RL. Using only 6K training examples and modest compute, RL improves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4%, a 14.6-point absolute gain. We release the complete workflow and training scripts to facilitate reproducible research on harnessed agentic RL.