学会预见：揭示同策略蒸馏的效率释放

摘要

在线策略蒸馏（OPD）已成为大语言模型的一种高效后训练范式。然而，现有研究大多将其优势归因于更密集、更稳定的监督信号，而OPD效率背后的参数级机制仍未被充分理解。本研究提出，OPD的高效性源于一种“前瞻性”特性：它在训练早期就为最终模型建立了稳定的更新轨迹。这种前瞻性体现在两个层面：首先，在模块分配层面，OPD能够识别边际效用较低的区域，并将更新聚焦于对推理更关键的模块上；其次，在更新方向层面，OPD展现出更强的低秩集中性，其主导子空间在训练早期便与最终更新子空间高度对齐。基于这些发现，我们提出了EffOPD——一种即插即用的加速方法，通过自适应选择外推步长并沿当前更新方向移动来加速OPD。EffOPD无需额外可训练模块或复杂超参数调优，在保持相当最终性能的同时，实现了平均3倍的训练加速。总体而言，我们的研究为理解OPD的高效性提供了参数动力学视角，并为设计更高效的大语言模型后训练方法提供了实践洞见。

English

On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models. However, existing studies largely attribute this advantage to denser and more stable supervision, while the parameter-level mechanisms underlying OPD's efficiency remain poorly understood. In this work, we argue that OPD's efficiency stems from a form of ``foresight'': it establishes a stable update trajectory toward the final model early in training. This foresight manifests in two aspects. First, at the Module-Allocation Level, OPD identifies regions with low marginal utility and concentrates updates on modules that are more critical to reasoning. Second, at the Update-Direction Level, OPD exhibits stronger low-rank concentration, with its dominant subspaces aligning closely with the final update subspace early in training. Building on these findings, we propose EffOPD, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction. EffOPD requires no additional trainable modules or complex hyperparameter tuning, and achieves an average training acceleration of 3times while maintaining comparable final performance. Overall, our findings provide a parameter-dynamics perspective for understanding the efficiency of OPD and offer practical insights for designing more efficient post-training methods for large language models.