SEED：面向智能体强化学习的自我进化在线策略蒸馏

SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

July 16, 2026

作者: Jinyang Wu, Shuo Yang, Zhengxi Lu, Fan Zhang, Yuhao Shen, Lang Feng, Haoran Luo, Zheng Lian, Shuai Zhang, Zhengqi Wen, Jianhua Tao

cs.AI

摘要

大型语言模型正越来越多地被训练成交互式智能体，用于处理涉及多轮交互、工具使用和环境反馈的长期任务。基于结果的强化学习提供了一种实用的优化范式，但其稀疏的轨迹级奖励对中间决策提供的指导有限，导致回合级结果与令牌级策略学习之间存在监督差距。我们提出SEED（自我进化的在轨策略蒸馏），这是一个自我进化的框架，它将已完成的在轨轨迹转化为训练时的后见技能，并将其行为效果蒸馏回策略模型中。SEED首先微调策略，使其分析已完成的轨迹并生成自然语言技能，这些技能捕捉可重用的工作流、关键观察或失败规避规则。在强化学习过程中，当前策略既收集轨迹，又充当分析器从中提取后见技能。因此，策略更新同时改进了后续决策和技能分析，使得后见监督能够随策略共同进化。SEED随后在普通和技能增强的背景下对采样动作重新评分，将技能引起的概率转移转化为密集的令牌级在轨蒸馏信号。该信号与基于结果的强化学习联合优化，确保辅助监督与当前轨迹分布保持一致。在基于文本和基于视觉的智能体任务上的大量实验表明，SEED持续提升了性能和样本效率，并展现出对未见场景的稳健泛化能力。我们的代码可在 https://github.com/jinyangwu/SEED 获取。

English

Large language models are increasingly trained as interactive agents for long-horizon tasks involving multi-turn interaction, tool use, and environment feedback. Outcome-based reinforcement learning (RL) provides a practical optimization paradigm, but its sparse trajectory-level rewards offer limited guidance on intermediate decisions, leaving a supervision gap between episode-level outcomes and token-level policy learning. We propose SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills and distills their behavioral effect back into the policy model. SEED first fine-tunes the policy to analyze completed trajectories and generate natural-language skills that capture reusable workflows, decisive observations, or failure-avoidance rules. During RL, the current policy both collects trajectories and serves as the analyzer that extracts hindsight skills from them. Policy updates therefore improve subsequent decision making and skill analysis together, allowing hindsight supervision to evolve with the policy. SEED then re-scores the sampled actions under ordinary and skill-augmented contexts, converting the skill-induced probability shift into a dense token-level on-policy distillation signal. This signal is jointly optimized with outcome-based RL, keeping the auxiliary supervision aligned with the current trajectory distribution. Extensive experiments on text-based and vision-based agentic tasks show that SEED consistently improves performance and sample efficiency, exhibiting robust generalization to unseen scenarios. Our code is available at https://github.com/jinyangwu/SEED.