论在线策略蒸馏的几何结构

摘要

在线策略蒸馏（OPD）被越来越多地用于提升大型语言模型的推理能力，但其训练动态仍未被充分理解。我们在参数空间中刻画了OPD更新的轨迹，并将其与监督微调（SFT）和可验证奖励强化学习（RLVR）进行了比较。一系列参数空间诊断始终将OPD置于一个宽松的非主成分区域：与SFT相比，其更新影响更少的权重，且更强烈地避开主方向；而与RLVR相比，其约束则相对宽松。除静态局部化外，OPD还表现出子空间锁定现象：其累积更新迅速进入一个狭窄的低维通道。将训练限制在训练早期形成的更新子空间内，可保持OPD的性能，但会显著降低SFT的效果，表明该锁定子空间对OPD具有功能上的充分性。控制实验进一步显示，稀疏化更新token或将rollout生成移至离线策略不会改变秩动态，而将OPD目标与RLVR混合则会改变它们。总体而言，这些结果表明OPD并非仅仅是SFT与RLVR之间的中间点，而是在参数空间中诱导出自身独特的更新几何结构。

English

On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood. We characterize the trajectory of OPD updates in parameter space and compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). A suite of parameter-space diagnostics consistently places OPD in a relaxed off-principal regime: compared with SFT, its updates affect fewer weights and avoid principal directions more strongly, while compared with RLVR, they remain less tightly constrained. Beyond this static localization, OPD exhibits subspace locking: its cumulative updates rapidly enter a narrow low-dimensional channel. Constraining training to the update subspace formed early in training preserves OPD performance but substantially degrades SFT, indicating that the locked subspace is functionally sufficient for OPD. Control experiments further show that sparsifying the update tokens and shifting rollout generation off-policy preserve the rank dynamics, whereas mixing the OPD objective with RLVR changes them. Overall, these results suggest that OPD is not merely an intermediate point between SFT and RLVR, but induces its own update geometry in parameter space.