潜空间在策略自蒸馏

摘要

使智能体能够从经验中学习并将其内化到策略中，已成为自演化人工智能的核心问题。在策略自蒸馏（OPSD）通过使用特权自教师对学生自身轨迹提供密集监督，为这一问题提供了有效途径；然而，现有方法仍严重依赖设计者指定的特权产物（如答案、反馈、技能或轨迹），限制了持续自我改进所需的端到端可学习性与可扩展性。在本工作中，我们提出潜在在策略自蒸馏（LOPD）。与提出另一种规定新特权上下文形式的手工设计OPSD变体不同，LOPD使教师自身的特权上下文可从经验中进行端到端学习。技术上，LOPD检索相关经验并将其组合为连续的潜在标记，作为自教师的条件输入；同时，学生从任务与交互历史中生成轨迹，并在每个访问过的前缀处接收密集的标记级监督。我们进一步引入特权间隔目标，以稳定和规范潜在上下文的学习。实验表明，LOPD展现出（I）强大的性能：在智能体工具使用与代码生成任务中均优于RLVR以及OPSD、SDPO和Skill-SD等代表性OPSD方法；（II）高学习效率：以不足GRPO和Skill-SD 30%的轨迹采样预算超越两者。消融研究进一步提供了直接证据，表明使特权上下文可学习是实现上述增益的必要条件。综上，这些结果将LOPD定位为迈向更可扩展、更自主的智能体进化范式的一步。

English

Enabling agents to learn from experience and internalize it into their policy has become a central problem in self-evolving AI. On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student's own trajectories; however, existing methods still rely heavily on designer-specified privileged artifacts (e.g., answers, feedback, skills, or trajectories), limiting the end-to-end learnability and scalability required for continual self-improvement. In this work, we introduce Latent On-Policy Self-Distillation (LOPD), which, rather than proposing another hand-crafted OPSD variant with a newly prescribed form of privileged context, makes the teacher's privileged context itself learnable end-to-end from experience. Technically, LOPD retrieves relevant experiences and composes them into continuous latent tokens that condition a self-teacher, while the student generates trajectories from the task and interaction history and receives dense token-level supervision at every visited prefix. We further introduce a privileged-margin objective to stabilize and regulate the learning of latent context. Empirically, LOPD demonstrates (I) strong performance, outperforming RLVR and representative OPSD methods including OPSD, SDPO, and Skill-SD across both agentic tool use and code generation; and (II) high learning efficiency, surpassing GRPO and Skill-SD with less than 30% of their rollout budget. Ablation studies further provide direct evidence that making privileged context learnable is necessary for realizing these gains. Together, these results position LOPD as a step toward a more scalable and self-directed paradigm for agent evolution.