当特权指导错位时：面向多轮智能体的状态匹配路由与情境化自蒸馏

摘要

特权在策略蒸馏通过允许同步教师利用仅训练可用的参考（例如成功轨迹）在每个回合对学生响应进行重新评分，为多轮智能体提供密集监督。然而，在交互式环境中，学生先前采取的动作会持续改变执行状态。当学生采取不同动作或以不同顺序完成子目标时，其轨迹展开可能到达参考未覆盖的状态，使得参考对于实际到达状态而言成为不可靠的指导来源。因此，不加区分地应用特权蒸馏会造成状态-参考不匹配。这种不匹配引出了一个核心目标：提供与学生当前执行状态兼容的特权参考指导。我们提出了状态匹配路由与情境化自蒸馏（SMRC-SD），它显式地确定特权轨迹应在何时以及如何指导在策略学生。在每个回合，SMRC-SD 都会验证学生当前执行状态是否与参考轨迹上的某个受支持状态匹配；仅在匹配状态下应用蒸馏，从而过滤掉参考缺乏局部兼容指导的回合。对于每个匹配状态，SMRC-SD 进一步从成功轨迹构建状态条件的教师上下文，使监督基于实际到达的状态。在 ALFWorld 和 WebShop 上，SMRC-SD 始终优于无条件的成功全路径蒸馏。使用 Qwen3-1.7B，它在 ALFWorld 上将任务成功率从 0.746 提升到 0.865，在 WebShop 上从 0.574 提升到 0.693。受控路由与上下文消融实验表明，选择局部受支持的回合和构建状态兼容的教师上下文都是实现这些提升的贡献因素。代码可在 https://github.com/liujunzhuo/SMRC-SD 获取。

English

Privileged on-policy distillation provides dense supervision for multi-turn agents by allowing a synchronized teacher to re-score the student's response at every turn with access to training-only references, such as successful trajectories. In interactive environments, however, the student's preceding actions continually change the execution state. As the student takes different actions or completes subgoals in a different order, its rollout may reach states not covered by the reference, making the reference an unreliable source of guidance for the state actually reached. Applying privileged distillation indiscriminately therefore creates state--reference mismatch. This mismatch motivates a central objective: providing privileged reference guidance that remains compatible with the student's current execution state. We introduce State-Matched Routing and Contextualized Self-Distillation (SMRC-SD), which explicitly determines when and how a privileged trajectory should guide an on-policy student. At each turn, SMRC-SD verifies whether the student's current execution state matches a supported state along the reference trajectory. Distillation is applied only at matched states, filtering out turns for which the reference lacks locally compatible guidance. For each matched state, SMRC-SD further constructs state-conditioned teacher context from the successful trajectory, grounding supervision in the state actually reached. Across ALFWorld and WebShop, SMRC-SD consistently outperforms unconditional successful full-path distillation. With Qwen3-1.7B, it improves task success from 0.746 to 0.865 on ALFWorld and from 0.574 to 0.693 on WebShop. Controlled routing and context ablations support both selecting locally supported turns and constructing state-compatible teacher context as contributors to these gains. Code is available at https://github.com/liujunzhuo/SMRC-SD.