MOPD：面向大语言模型后训练能力整合的多教师同策略蒸馏

摘要

现代大型语言模型（LLMs）在后期训练中依赖强化学习来提升特定能力，但将多种能力整合到单一模型中仍面临困难。现有方法如离线策略微调（Off-Policy Finetune）和混合强化学习（Mix-RL）要么效率低下，要么会导致性能损失。本文提出多教师在线策略蒸馏（MOPD），这是一种用于整合多个领域强化学习教师能力的后期训练范式：我们首先针对每个领域进行专门的强化学习训练，获得一组领域教师，然后在学生自身的轨迹中将这些教师蒸馏到学生模型中。这消除了暴露偏差，并提供了密集的优化信号。在Qwen3-30B-A3B模型上，MOPD在性能上超越了混合强化学习、级联强化学习、离线策略微调和参数合并等基线方法，几乎继承了每位教师的所有能力。MOPD还支持领域教师的并行独立开发，消除了多领域后期训练中典型的跨领域耦合问题。MOPD已部署在工业级前沿模型MiMo-V2-Flash的后期训练中，验证了其在整合前沿LLM能力方面的实用价值。

English

Modern large language models (LLMs) rely on reinforcement learning during post-training to push specific capabilities, yet integrating multiple capabilities into one model remains hard. Existing methods, such as Off-Policy Finetune and Mix-RL, are either inefficient or lose performance. In this work, we propose Multi-teacher On-Policy Distillation (MOPD), a post-training paradigm for combining the capabilities of multiple domain RL teachers: we first run per-domain specialised RL to obtain a set of domain teachers, then distill these teachers into the student on its own rollouts. This eliminates exposure bias and provides a dense optimization signal. On Qwen3-30B-A3B, MOPD outperforms Mix-RL, Cascade RL, Off-Policy Finetune, and Param-Merge baselines, inheriting nearly all of each teacher's capability. MOPD also enables parallel, independent development of domain teachers, removing the cross-domain coupling typical of multi-domain post-training. MOPD has been deployed in the post-training of MiMo-V2-Flash, an industrial-scale frontier model, demonstrating its practical value for capability integration in frontier-scale LLMs.