AsyncOPD：在策略蒸馏可以有多陈旧？

摘要

同策略蒸馏（OPD）通过引导学生在其自身轨迹上依据教师反馈进行训练，在大语言模型（LLM）后训练中日益重要。然而，与强化学习（RL）类似，OPD面临同策略系统瓶颈——对于推理任务，轨迹生成可能占据训练时间的主导地位。异步训练流水线通过将轨迹生成与学习器更新解耦可缓解该瓶颈，但这会引入陈旧策略数据。虽然已有工作研究了异步RL中的陈旧数据问题，但其在OPD中的影响仍鲜有探索。本文首次系统研究异步OPD中的陈旧性问题，重点关注一种实际场景：教师反馈通过局部KL散度实现，且全词汇教师logits的存储或传输成本过高，需使用有限教师评分缓存。我们首先证明，KL散度方向会改变陈旧数据问题的性质：教师加权前向KL对陈旧轨迹更鲁棒，而学生加权反向KL则较脆弱。其次，针对这种脆弱的反向KL情况，我们考察了为稳定异步RL而设计的方法能否缓解OPD的陈旧性。实验中，这些方法并未优于更简单的OPD专用替代方案——在学习器端利用当前学生重新计算反向KL信号。第三，我们分析了有限教师评分缓存如何对稀疏和采样反向KL OPD估计器产生偏差-方差权衡，这推动采用多样本蒙特卡洛（MC）方法——在降低单样本方差的同时保持MC可纠偏特性。最后，我们提出并开源AsyncOPD——基于上述估计器选择构建的完全异步OPD训练流水线。实验表明，AsyncOPD将训练吞吐量提升至严格同步训练的1.6至3.8倍，同时达到可比精度。

English

On-policy distillation (OPD) trains a student on its own rollouts guided by teacher feedback and is becoming increasingly important for large language model (LLM) post-training. Like reinforcement learning (RL), however, OPD faces an on-policy systems bottleneck, as rollouts can dominate training time for reasoning workloads. Asynchronous training pipelines can alleviate this bottleneck by decoupling rollout generation from learner updates, but doing so introduces stale-policy data. While prior work has studied stale data in asynchronous RL, its effects in OPD remain underexplored. We present the first systematic study of staleness in asynchronous OPD, focusing on a practical setting where teacher feedback is implemented through local KL losses and full-vocabulary teacher logits are too expensive to store or transfer, necessitating finite teacher-score caches. We first show that KL direction changes the stale-data problem: teacher-weighted forward KL is more robust to stale rollouts, whereas student-weighted reverse KL is vulnerable. Second, for this vulnerable reverse-KL case, we study whether methods designed to stabilize asynchronous RL can mitigate OPD staleness. In our experiments, they do not improve over a simpler OPD-specific surrogate: recomputing the reverse-KL signal under the current student at learner time. Third, we analyze how finite teacher-score caches create a bias-variance tradeoff for sparse and sampled reverse-KL OPD estimators. This motivates multi-sample Monte Carlo (MC), which preserves MC correctability while reducing one-sample variance. Finally, we present and open-source AsyncOPD, a fully asynchronous OPD training pipeline built from these estimator choices. Experiments show that AsyncOPD improves training throughput by 1.6times to 3.8times over strict synchronous training while reaching comparable accuracy.