論同策略蒸餾的幾何

摘要

同策略蒸餾（OPD）日益用於提升大型語言模型的推理能力，但其訓練動態仍缺乏深入理解。我們描述了OPD在參數空間中的更新軌跡，並將其與監督微調（SFT）及帶有可驗證獎勵的強化學習（RLVR）進行比較。一系列參數空間診斷方法一致顯示，OPD處於一種寬鬆的非主成分（off-principal）區間：相較於SFT，其更新影響較少的權重，且更強烈地避開主成分方向；而相較於RLVR，其更新所受約束則較為寬鬆。除了這種靜態局部性，OPD還表現出子空間鎖定（subspace locking）現象：其累積更新迅速進入狹窄的低維通道。將訓練限制於訓練早期形成的更新子空間內，可保持OPD的性能，但會顯著降低SFT的性能，表明該鎖定子空間對OPD而言具有功能充分性。控制實驗進一步顯示：稀疏化更新令牌以及將滾動生成移為離策略，並不會改變秩動態，而將OPD目標與RLVR混合則會改變之。整體而言，這些結果表明，OPD不僅僅是介於SFT與RLVR之間的中間點，而是在參數空間中誘發了其獨有的更新幾何結構。

English

On-policy distillation (OPD) is increasingly used to improve large language model reasoning, but its training dynamics remain poorly understood. We characterize the trajectory of OPD updates in parameter space and compare it with supervised fine-tuning (SFT) and reinforcement learning with verifiable rewards (RLVR). A suite of parameter-space diagnostics consistently places OPD in a relaxed off-principal regime: compared with SFT, its updates affect fewer weights and avoid principal directions more strongly, while compared with RLVR, they remain less tightly constrained. Beyond this static localization, OPD exhibits subspace locking: its cumulative updates rapidly enter a narrow low-dimensional channel. Constraining training to the update subspace formed early in training preserves OPD performance but substantially degrades SFT, indicating that the locked subspace is functionally sufficient for OPD. Control experiments further show that sparsifying the update tokens and shifting rollout generation off-policy preserve the rank dynamics, whereas mixing the OPD objective with RLVR changes them. Overall, these results suggest that OPD is not merely an intermediate point between SFT and RLVR, but induces its own update geometry in parameter space.