Agent Lightning v1.0：迈向受控的智能体强化学习

摘要

现代智能体运行在管理工具、上下文和控制流的智能体框架（agent harness）内部，这使得框架成为智能体系统的关键组成部分。我们最初的Agent Lightning引入了一种解耦架构，通过大语言模型端点代理将任意智能体与强化学习训练相连接，该方法后来被verl Uni-Agent、AReaL 2.0、slime和Polar等框架所采用。我们将这一范式称为基于框架的智能体强化学习（harnessed agentic RL），其中部署时的框架直接参与模型的后训练过程。基于框架的智能体强化学习与传统智能体强化学习存在根本性差异：由框架而非训练引擎持有环境交互循环，而训练器仅观察大语言模型请求-响应对的序列。这引入了重新分词、样本合并、优势计算、损失归一化和后端调度等方面的挑战，这些挑战可能对训练稳定性和有效性产生重大影响。我们提出Agent Lightning v1.0，一个以约3,500行代码实现的轻量级基于框架的智能体强化学习框架。它支持任意智能体框架，并可作为研究上述挑战的实用测试平台。我们在指令遵循、搜索和编码智能体上对其进行了评估，并提供了完整的、可复现的编码智能体强化学习流水线。仅使用6K训练样本和适度的计算资源，强化学习将Qwen3.5-9B在SWE-bench Verified上的性能从41.8%提升至56.4%，实现了14.6个百分点的绝对提升。我们发布了完整的工作流程和训练脚本，以促进基于框架的智能体强化学习的可复现研究。

English

Modern agents operate inside agent harnesses that manage tools, context, and control flow, making the harness a critical part of the agent system. Our original Agent Lightning introduced a disaggregated architecture that connects arbitrary agents to RL training through an LLM endpoint proxy, an approach later adopted by frameworks such as verl Uni-Agent, AReaL 2.0, slime, and Polar. We refer to this paradigm as harnessed agentic RL, where the deploy-time harness directly participates in model post-training. Harnessed agentic RL differs fundamentally from traditional agentic RL: the harness, rather than the training engine, owns the environment interaction loop, while the trainer observes only sequences of LLM request-response pairs. This introduces challenges in retokenization, sample merging, advantage calculation, loss normalization, and backend scheduling, which can substantially affect training stability and effectiveness. We present Agent Lightning v1.0, a lightweight framework for harnessed agentic RL implemented in approximately 3,500 lines of code. It supports arbitrary agent harnesses and serves as a practical testbed for studying these challenges. We evaluate it on instruction-following, search, and coding agents, and provide a complete reproducible pipeline for coding-agent RL. Using only 6K training examples and modest compute, RL improves Qwen3.5-9B on SWE-bench Verified from 41.8% to 56.4%, a 14.6-point absolute gain. We release the complete workflow and training scripts to facilitate reproducible research on harnessed agentic RL.