D^3-MOPD：面向高效多教師蒸餾的自適應動態領域排程

摘要

多教師在策略蒸餾（MOPD）透過最小化學生自身軌跡上的逐領域反向KL散度，將數個領域專家教師蒸餾至單一學生。現有方法通常會在訓練前固定逐領域數據混合比例，卻忽略了不同領域的收斂速度差異極大：有些領域早期即趨於停滯，有些則在整個訓練預算期間持續提升。因此，固定的混合比例會將計算資源浪費在快速收斂的領域上，並對收斂較慢的領域訓練不足。為了解決此問題，我們提出了 D^3-MOPD（針對MOPD的動態領域調度），這是一種零開銷的調度器，它重新利用訓練過程中已產生的逐領域反向KL信號，在線調整領域混合比例。透過在訓練進程之外異步運行，一個進程外監視器會定期追蹤每個領域的KL軌跡，估算剩餘提升空間與當前改進速率，並相應調整領域採樣比例，而無需修改核心訓練迴圈。我們的 D^3-MOPD 能自然擴展至任意數量的領域，且隨著更多領域引入更多樣化的收斂模式供調度器利用，預期收益也隨之增加。在一個從四個領域專家教師蒸餾而來的 Qwen3.6-35B-A3B 學生上，D^3-MOPD 縮小了學生與教師平均性能差距的 97%，而原始 MOPD 僅縮小 63%；它還以約三分之一的軌跡步數達到相同的峰值性能，並在七個基準測試中的三個上超越了專家教師。

English

Multi-teacher on-policy distillation (MOPD) distills several domain-expert teachers into a single student by minimizing per-domain reverse-KL divergence on the student's own rollouts. Existing approaches typically fix the per-domain data mixture before training, overlooking the fact that different domains converge at substantially different rates: some plateau early while others continue to improve throughout the training budget. A fixed mixture therefore wastes compute on fast-converging domains and undertrains slower-converging ones. To address this, we propose D^3-MOPD (Dynamic Domain ScheDuling for MOPD), a zero-overhead scheduler that repurposes the per-domain reverse-KL signal already produced during training to adapt the domain mixture online. Running asynchronously outside the training process, an off-process watcher periodically tracks each domain's KL trajectory, estimates remaining headroom and current improvement rate, and accordingly adjusts the domain sampling ratios without altering the core training loop. Our D^3-MOPD scales naturally to arbitrary numbers of domains, and the expected benefit grows as more domains introduce more diverse convergence patterns for the scheduler to exploit. On a Qwen3.6-35B-A3B student distilled from four domain-expert teachers, D^3-MOPD closes 97% of the average student-to-teacher performance gap, compared with 63% for vanilla MOPD, reaches the same peak performance with an approximately 3times reduction in rollout steps, and surpasses the specialist teachers on three of seven benchmarks.