OPID：面向智能體強化學習的在線策略技能蒸餾

摘要

基於結果的強化學習為語言智能體提供了穩定的優化基礎，但其稀疏的軌跡級獎勵對於哪些中間決策應被強化或抑制提供的引導有限。線上策略自我蒸餾提供了密集的詞元級監督，然而現有的技能條件變體通常依賴於外部技能記憶庫或檢索到的特權上下文，這些不僅維護成本高昂，還可能與當前策略在多輪互動中產生的狀態分佈不匹配。我們提出 OPID（線上策略技能蒸餾），這是一個直接從已完成的線上策略軌跡中提取技能監督的框架。OPID 將軌跡事後視角表示為層級式技能：回合級技能捕捉全域工作流程或失敗規避規則，而步驟級技能則在關鍵時間步捕捉局部決策知識。關鍵優先路由機制在識別出關鍵決策時使用步驟級技能，否則預設以回合級技能作為引導。所選技能被注入互動歷史中，使舊策略能夠同時在原始語境與技能增強的語境下重新評分相同的取樣回應。由此產生的對數概率偏移形成了詞元級自我蒸餾優勢，並與結果優勢結合用於策略優化。因此 OPID 保留了強化學習作為主要訓練目標，同時引入了密集且與分佈匹配的事後監督。在 ALFWorld、WebShop 和基於搜尋的問答實驗中，OPID 在智能體性能、樣本效率與穩健性方面普遍優於僅使用結果的強化學習和現有技能蒸餾基準。我們的代碼已公開於 https://github.com/jinyangwu/OPID/tree/main。

English

Outcome-based reinforcement learning provides a stable optimization backbone for language agents, but its sparse trajectory-level rewards provide little guidance on which intermediate decisions should be reinforced or suppressed. On-policy self-distillation offers dense token-level supervision, yet existing skill-conditioned variants often rely on external skill memories or retrieved privileged context, which are costly to maintain and can be mismatched with the state distribution induced by the current policy in multi-turn interaction. We propose OPID (On-Policy Skill Distillation), a framework that extracts skill supervision directly from completed on-policy trajectories. OPID represents trajectory hindsight as hierarchical skills: episode-level skills capture global workflows or failure-avoidance rules, while step-level skills capture local decision knowledge at critical timesteps. A critical-first routing mechanism uses step-level skills when critical decisions are identified and falls back to episode-level skills as default guidance otherwise. The selected skill is injected into the interaction history, allowing the old policy to re-score the same sampled response under both original and skill-augmented contexts. The resulting log-probability shift yields a token-level self-distillation advantage, which is combined with the outcome advantage for policy optimization. OPID thus preserves RL as the primary training objective while introducing dense, distribution-matched hindsight supervision. Experiments on ALFWorld, WebShop and Search-based QA demonstrate that OPID generally improves agent performance, sample efficiency, and robustness over outcome-only RL and existing skill-distillation baselines. Our code is available at https://github.com/jinyangwu/OPID/tree/main.