DOPD：雙重在策略蒸餾

摘要

同策蒸馏（OPD）通过密集的词元级信号监督学生模型采样的轨迹，实现了更优的能力迁移。为提供高质量的监督源以进一步提升蒸馏性能边界，一个直观方向是将先验信息注入教师或学生模型本身。然而，这种额外输入会引发一种潜在失败模式——我们称之为“先验幻觉”：即学生本应弥补的可迁移能力差距与只能模仿却无法复现的信息不对称差距被混为一谈。该问题因词元级监督固有的非均匀性而进一步加剧——仅少数关键词元携带具有决定性能力表征的信号。为此，我们提出DOPD——一种优势感知的双重蒸馏范式，该范式基于教师先验策略与学生先验策略之间的优势差距和相对概率，动态路由词元级监督路径。每个词元会从教师或学生自身接收不同强度、目标与策略的监督信号，在传递可信能力的同时获取辅助信号，从而缓解先验幻觉。在大语言模型（LLM）与视觉语言模型（VLM）设定下的广泛实验表明，DOPD始终优于标准同策蒸馏及其他对比方法。稳定性、鲁棒性、持续学习与分布外任务上的进一步结果验证了其优越性。

English

On-policy distillation (OPD) offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals. To furnish high-quality supervision sources and thereby elevate the performance frontier of distillation, an intuitive direction is to infuse privileged information to either teacher or student itself. However, this additional input induces a potential failure mode we dub privilege illusion: a pattern that conflates the transferable capability gap that students are meant to close, and the information asymmetry gap that can only be mimicked but never replicated. This issue is further amplified by the inherent non-uniformity of token-level supervision, where only a small subset of tokens carries pivotal capability-bearing signals. To this end, we propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies based on their advantage gap and relative probabilities. Each token receives supervision of different strength, objective, and strategy from either teacher or student itself, which transfers credible capability while simultaneously receiving auxiliary signals, to alleviate privilege illusion. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD and other counterparts. Further results on stability, robustness, continual learning, and out-of-distribution tasks validate its superiority.