DOPD：双在线策略蒸馏

摘要

在策略蒸馏（OPD）通过密集的token级信号监督学生采样轨迹，实现了更优的能力迁移。为了提供高质量的监督源并进一步提升蒸馏的性能边界，一个直观的方向是向教师或学生自身注入特权信息。然而，这种额外输入会引发一种潜在的失败模式，我们称之为“特权幻觉”：这种模式混淆了学生应该弥合的可迁移能力差距与只能模仿而无法复制的信息不对称差距。这一问题又被token级监督固有的非均匀性进一步放大——即只有一小部分token承载着关键的能力信号。为此，我们提出DOPD，一种优势感知的双重蒸馏范式，它基于优势差距和相对概率，在特权教师策略与特权学生策略之间动态路由token级监督。每个token从教师或学生自身接收不同强度、目标和策略的监督，在转移可信能力的同时接收辅助信号，以缓解特权幻觉。在大型语言模型（LLM）和视觉-语言模型（VLM）上的大量实验表明，DOPD始终优于普通OPD及其他对比方法。在稳定性、鲁棒性、持续学习和分布外任务上的进一步结果验证了其优越性。

English

On-policy distillation (OPD) offers superior capacity transfer by supervising student-sampled trajectories with dense token-level signals. To furnish high-quality supervision sources and thereby elevate the performance frontier of distillation, an intuitive direction is to infuse privileged information to either teacher or student itself. However, this additional input induces a potential failure mode we dub privilege illusion: a pattern that conflates the transferable capability gap that students are meant to close, and the information asymmetry gap that can only be mimicked but never replicated. This issue is further amplified by the inherent non-uniformity of token-level supervision, where only a small subset of tokens carries pivotal capability-bearing signals. To this end, we propose DOPD, an advantage-aware dual distillation paradigm that dynamically routes token-level supervision between privileged teacher and privileged student policies based on their advantage gap and relative probabilities. Each token receives supervision of different strength, objective, and strategy from either teacher or student itself, which transfers credible capability while simultaneously receiving auxiliary signals, to alleviate privilege illusion. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that DOPD consistently outperforms Vanilla OPD and other counterparts. Further results on stability, robustness, continual learning, and out-of-distribution tasks validate its superiority.