Open-MOPD：多教师同策略蒸馏中能力失衡的诊断与修复

摘要

多教师同策略蒸馏（M-OPD）已成为一种有前景的范式，通过密集的词元级奖励监督，将领域专用的强化学习专家整合为单一的通才学生模型。尽管其在实践中取得了成功，但支配多教师能力整合的优化动力学仍未被充分理解，且缺乏开放、可严格复现的完整流程。在本研究中，我们在SmolLM3-3B-Base上建立了一个带有神谕路由的受控M-OPD基准，将能力整合与路由模糊性分离开来。我们的研究发现了一个显著的能力整合差距：相对于领域路由的神谕集成，标准M-OPD仅捕获了35.6%的可用性能空间，其中指令遵循等简洁任务遭受严重退化并过早陷入停滞。关键在于，我们证明这一失败并非源于梯度冲突，而是源于词元级优化预算的严重错配。这一病态现象由三个正交因素驱动：跨领域的结构性序列长度差异、非均匀学习率导致的动态收敛漂移，以及异步策略更新带来的多步奖励陈旧性。为解决这些不平衡问题，我们引入了Open-MOPD，一个包含词元份额平衡、差距感知的动态预算分配和学生奖励刷新的系统性框架。这些机制共同系统地恢复了跨领域平衡，将性能空间恢复率从35.6%提升至83.4%，且仅需单个可部署的学生模型。我们在学术可及的计算预算下，完全开源了端到端的后训练流程、训练轨迹和评估套件。

English

Multi-teacher on-policy distillation (M-OPD) has emerged as a promising paradigm for consolidating domain-specialized reinforcement learning (RL) experts into a single generalist student via dense, token-level reward supervision. Despite its practical success, the optimization dynamics governing multi-teacher capability integration remain poorly understood, and open, rigorously reproducible recipes are conspicuously lacking. In this work, we establish a controlled M-OPD benchmark on SmolLM3-3B-Base with oracle routing, isolating capability integration from routing ambiguity. Our investigation reveals a pronounced capability integration gap: standard M-OPD captures only 35.6% of the available headroom relative to a domain-routed oracle ensemble, with concise tasks such as instruction following suffering severe degradation and premature stagnation. Crucially, we show that this failure stems not from gradient conflict, but from a severe misallocation of the token-level optimization budget. This pathology is driven by three orthogonal factors: structural sequence-length disparities across domains, dynamic convergence drift due to non-uniform learning rates, and multi-step reward staleness from asynchronous policy updates. To resolve these imbalances, we introduce Open-MOPD, a principled framework incorporating token-share balancing, gap-aware dynamic budget allocation, and student reward refresh. Together, these mechanisms systematically restore cross-domain balance, elevating headroom recovery from 35.6% to 83.4% in a single deployable student. We fully open-source our end-to-end post-training recipe, training trajectories, and evaluation suites on an academically accessible hardware budget.