D^3-MOPD：面向高效多教师蒸馏的自适应动态领域调度

摘要

多教师在策略蒸馏（MOPD）通过最小化学生自身生成的轨迹上各领域的反向KL散度，将多个领域专家教师的知识蒸馏到单一学生模型中。现有方法通常在训练前固定各领域的数据混合比例，却忽视了不同领域收敛速度存在显著差异：有些领域早早进入平台期，而另一些领域在整个训练预算内持续提升。固定的混合比例因此既在快速收敛的领域上浪费算力，又对收敛较慢的领域训练不足。针对这一问题，我们提出D³-MOPD（面向MOPD的动态领域调度，Dynamic Domain ScheDuling for MOPD）——一种零开销调度器，直接复用训练过程中已产生的各领域反向KL信号，在线调整领域混合比例。调度器在训练进程之外异步运行，定期追踪各领域的KL轨迹，估算剩余提升空间与当前改进速率，并据此调整领域采样比例，而无需改动核心训练循环。D³-MOPD可自然扩展到任意数量的领域，且随着领域数量增加、收敛模式越发多样，调度器的预期收益也越大。在由四个领域专家教师蒸馏Qwen3.6-35B-A3B学生的实验中，D³-MOPD弥合了平均学生-教师性能差距的97%，而vanilla MOPD仅弥合63%；D³-MOPD在轨迹生成步数缩减约3倍的情况下达到相同峰值性能，并在七个基准中的三个上超越了专家教师。

English

Multi-teacher on-policy distillation (MOPD) distills several domain-expert teachers into a single student by minimizing per-domain reverse-KL divergence on the student's own rollouts. Existing approaches typically fix the per-domain data mixture before training, overlooking the fact that different domains converge at substantially different rates: some plateau early while others continue to improve throughout the training budget. A fixed mixture therefore wastes compute on fast-converging domains and undertrains slower-converging ones. To address this, we propose D^3-MOPD (Dynamic Domain ScheDuling for MOPD), a zero-overhead scheduler that repurposes the per-domain reverse-KL signal already produced during training to adapt the domain mixture online. Running asynchronously outside the training process, an off-process watcher periodically tracks each domain's KL trajectory, estimates remaining headroom and current improvement rate, and accordingly adjusts the domain sampling ratios without altering the core training loop. Our D^3-MOPD scales naturally to arbitrary numbers of domains, and the expected benefit grows as more domains introduce more diverse convergence patterns for the scheduler to exploit. On a Qwen3.6-35B-A3B student distilled from four domain-expert teachers, D^3-MOPD closes 97% of the average student-to-teacher performance gap, compared with 63% for vanilla MOPD, reaches the same peak performance with an approximately 3times reduction in rollout steps, and surpasses the specialist teachers on three of seven benchmarks.