當特權引導錯位：多輪智能體的狀態匹配路由與情境化自蒸餾

摘要

特權型同策略蒸餾透過允許同步教師模型在每一輪利用僅訓練可得的參考（例如成功軌跡）重新評分學生模型的回應，為多輪代理提供密集監督。然而，在互動環境中，學生模型先前的動作會持續改變執行狀態。當學生模型採取不同動作或依不同順序完成子目標時，其展開軌跡可能到達參考軌跡未涵蓋的狀態，使得參考對於實際到達的狀態成為不可靠的引導來源。因此，不加區別地應用特權蒸餾會造成狀態-參考不匹配。此不匹配促使一項核心目標：提供與學生模型當前執行狀態相容的特權參考引導。我們提出狀態匹配路由與情境化自蒸餾（State-Matched Routing and Contextualized Self-Distillation, SMRC-SD），明確決定特權軌跡應在何時以及如何引導同策略學生模型。在每一輪，SMRC-SD 驗證學生模型當前的執行狀態是否與參考軌跡中的受支援狀態匹配。蒸餾僅在匹配的狀態上應用，過濾掉參考缺乏局部相容引導的輪次。對於每個匹配的狀態，SMRC-SD 進一步從成功軌跡建構狀態條件化的教師上下文，將監督立足於實際到達的狀態。在 ALFWorld 與 WebShop 上，SMRC-SD 持續優於無條件的成功完整路徑蒸餾。使用 Qwen3-1.7B 時，SMRC-SD 在 ALFWorld 上將任務成功率從 0.746 提升至 0.865，在 WebShop 上從 0.574 提升至 0.693。受控的路由與上下文消融實驗顯示，選擇局部受支援的輪次與建構狀態相容的教師上下文兩者皆為這些增益的促成因素。程式碼已公開於 https://github.com/liujunzhuo/SMRC-SD。

English

Privileged on-policy distillation provides dense supervision for multi-turn agents by allowing a synchronized teacher to re-score the student's response at every turn with access to training-only references, such as successful trajectories. In interactive environments, however, the student's preceding actions continually change the execution state. As the student takes different actions or completes subgoals in a different order, its rollout may reach states not covered by the reference, making the reference an unreliable source of guidance for the state actually reached. Applying privileged distillation indiscriminately therefore creates state--reference mismatch. This mismatch motivates a central objective: providing privileged reference guidance that remains compatible with the student's current execution state. We introduce State-Matched Routing and Contextualized Self-Distillation (SMRC-SD), which explicitly determines when and how a privileged trajectory should guide an on-policy student. At each turn, SMRC-SD verifies whether the student's current execution state matches a supported state along the reference trajectory. Distillation is applied only at matched states, filtering out turns for which the reference lacks locally compatible guidance. For each matched state, SMRC-SD further constructs state-conditioned teacher context from the successful trajectory, grounding supervision in the state actually reached. Across ALFWorld and WebShop, SMRC-SD consistently outperforms unconditional successful full-path distillation. With Qwen3-1.7B, it improves task success from 0.746 to 0.865 on ALFWorld and from 0.574 to 0.693 on WebShop. Controlled routing and context ablations support both selecting locally supported turns and constructing state-compatible teacher context as contributors to these gains. Code is available at https://github.com/liujunzhuo/SMRC-SD.