潛在同策略自蒸餾

摘要

讓智慧體從經驗中學習並將其內化為策略，已成為自我演化AI的核心問題。同策略自我蒸餾（OPSD）透過利用特權自我教師對學生自身的軌跡提供密集監督，提供了一條有效途徑；然而，現有方法仍高度依賴設計者指定的特權產物（例如答案、回饋、技能或軌跡），限制了持續自我改進所需的端到端可學習性與可擴展性。在本研究中，我們提出潛在同策略自我蒸餾（LOPD），其並非提出另一種以新形式指定特權上下文的手工設計OPSD變體，而是讓教師的特權上下文本身能夠端到端地從經驗中學習。在技術上，LOPD檢索相關經驗並將其組合成連續的潛在令牌，用以條件化自我教師，同時學生從任務與互動歷史中生成軌跡，並在每個造訪的前綴處接收密集的令牌級監督。我們進一步引入特權間隔目標函數，以穩定並規範潛在上下文的學習。在實驗上，LOPD展現了（I）強大的性能，在智慧體工具使用與程式碼生成兩項任務中均優於RLVR及代表性的OPSD方法（包括OPSD、SDPO與Skill-SD）；以及（II）高學習效率，以不到GRPO與Skill-SD 30%的展開預算即超越兩者。消融研究進一步提供了直接證據，表明使特權上下文可學習對於實現這些增益是必要的。綜上所述，這些結果將LOPD定位為邁向更具可擴展性與自導向性的智慧體演化範式的一步。

English

Enabling agents to learn from experience and internalize it into their policy has become a central problem in self-evolving AI. On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student's own trajectories; however, existing methods still rely heavily on designer-specified privileged artifacts (e.g., answers, feedback, skills, or trajectories), limiting the end-to-end learnability and scalability required for continual self-improvement. In this work, we introduce Latent On-Policy Self-Distillation (LOPD), which, rather than proposing another hand-crafted OPSD variant with a newly prescribed form of privileged context, makes the teacher's privileged context itself learnable end-to-end from experience. Technically, LOPD retrieves relevant experiences and composes them into continuous latent tokens that condition a self-teacher, while the student generates trajectories from the task and interaction history and receives dense token-level supervision at every visited prefix. We further introduce a privileged-margin objective to stabilize and regulate the learning of latent context. Empirically, LOPD demonstrates (I) strong performance, outperforming RLVR and representative OPSD methods including OPSD, SDPO, and Skill-SD across both agentic tool use and code generation; and (II) high learning efficiency, surpassing GRPO and Skill-SD with less than 30% of their rollout budget. Ablation studies further provide direct evidence that making privileged context learnable is necessary for realizing these gains. Together, these results position LOPD as a step toward a more scalable and self-directed paradigm for agent evolution.