SEED：面向自主強化學習的自我演化同策略蒸餾

SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learning

July 16, 2026

作者: Jinyang Wu, Shuo Yang, Zhengxi Lu, Fan Zhang, Yuhao Shen, Lang Feng, Haoran Luo, Zheng Lian, Shuai Zhang, Zhengqi Wen, Jianhua Tao

cs.AI

摘要

大型語言模型正越來越被訓練為互動式代理，以處理涉及多輪互動、工具使用和環境反饋的長期任務。基於結果的強化學習提供了一種實用的優化典範，但其稀疏的軌跡層級獎勵對中間決策提供的指導有限，導致在回合層級結果與詞元層級策略學習之間存在監督差距。我們提出SEED（自我進化的同策略蒸餾），這是一個自我進化的框架，能將完成的同策略軌跡轉化為訓練時期的後見技巧，並將其行為效果蒸餾回策略模型中。SEED首先微調策略，以分析已完成的軌跡並生成自然語言技巧，這些技巧捕捉可重複使用的工作流程、關鍵觀察或避免失敗的規則。在強化學習期間，當前策略既收集軌跡，也充當從中提取後見技巧的分析器。因此，策略更新能同時改善後續的決策制定與技巧分析，使後見監督能隨著策略演進。然後，SEED在普通情境與技巧增強情境下重新評分取樣的動作，將技巧引起的機率偏移轉化為密集的詞元層級同策略蒸餾信號。該信號與基於結果的強化學習共同優化，使輔助監督與當前軌跡分布保持一致。在基於文本與基於視覺的代理任務上的大量實驗表明，SEED持續提升性能與樣本效率，並展現出對未見過場景的穩健泛化能力。我們的程式碼可在 https://github.com/jinyangwu/SEED 取得。

English

Large language models are increasingly trained as interactive agents for long-horizon tasks involving multi-turn interaction, tool use, and environment feedback. Outcome-based reinforcement learning (RL) provides a practical optimization paradigm, but its sparse trajectory-level rewards offer limited guidance on intermediate decisions, leaving a supervision gap between episode-level outcomes and token-level policy learning. We propose SEED (SElf-Evolving On-Policy Distillation), a self-evolving framework that converts completed on-policy trajectories into training-time hindsight skills and distills their behavioral effect back into the policy model. SEED first fine-tunes the policy to analyze completed trajectories and generate natural-language skills that capture reusable workflows, decisive observations, or failure-avoidance rules. During RL, the current policy both collects trajectories and serves as the analyzer that extracts hindsight skills from them. Policy updates therefore improve subsequent decision making and skill analysis together, allowing hindsight supervision to evolve with the policy. SEED then re-scores the sampled actions under ordinary and skill-augmented contexts, converting the skill-induced probability shift into a dense token-level on-policy distillation signal. This signal is jointly optimized with outcome-based RL, keeping the auxiliary supervision aligned with the current trajectory distribution. Extensive experiments on text-based and vision-based agentic tasks show that SEED consistently improves performance and sample efficiency, exhibiting robust generalization to unseen scenarios. Our code is available at https://github.com/jinyangwu/SEED.