解密同策略蒸餾：角色、病理與調控

摘要

在線策略蒸餾（OPD）已成為大型語言模型後訓練的關鍵範式，但其訓練動態仍未被充分理解。我們進行了一項系統性研究，探討OPD的角色、病理現象及調節機制。首先，我們釐清OPD作為探索催化劑的角色：它透過密集的詞元層級引導，將學生模型導向正確的推理路徑，而不擴展能力上限。我們透過證明提示多樣性比每問題採樣數量更重要，以及關鍵的是，OPD的有效性完全取決於其引導訊號的品質，來確認這一點。此依賴性暴露了兩種破壞探索的病理現象。師生模型不匹配發生於大型教師-學生分佈差距導致引導訊號與任務正確性不一致，使探索轉向反效果的方向。長度利用則源於聚合的詞元層級目標產生與長度相關的捷徑，使學生模型能透過回應截斷或冗餘填充來操縱獎勵景觀，從而探索退化長度模式而非推理策略。為控制這些病理現象，我們研究輕量級訊號調節機制：優勢裁剪與對數尺度壓縮，確保探索由可靠訊號引導。在七個基準測試上的實驗表明，這些調節機制能減輕長度利用並實現有效蒸餾，穩定超越OPD變體與RLVR基線，從而確認良好調節的訊號品質（而非單純的教師模型規模）是OPD中成功探索的關鍵。

English

On-policy distillation (OPD) has become a key paradigm in LLM post-training, yet its training dynamics remain poorly understood. We present a systematic study examining the role, pathologies, and regulations of OPD. We first clarify the role of OPD as an exploration catalyst: it steers the student toward correct reasoning paths via dense token-level guidance, without expanding capability ceiling. We confirm this by showing that prompt diversity matters more than per-problem sampling numbers, and critically, that the effectiveness of OPD hinges entirely on the quality of its guiding signal. This dependency exposes two pathologies that derail exploration. The Student-Teacher Mismatch occurs when a large teacher-student distributional gap causes the guiding signal to misalign with task correctness, steering exploration in counterproductive directions. Length Exploitation arises when the aggregated token-level objective creates length-dependent shortcuts, allowing the student to game the reward landscape through response truncation or redundant padding, exploring degenerate length modes rather than reasoning strategies. To tame these pathologies, we investigate lightweight signal regulations: advantage clipping and log-scale compression, ensuring exploration is guided by faithful signals. Experiments across seven benchmarks demonstrate that these regulations alleviate length exploitation and enable effective distillation, stably surpassing OPD variants and RLVR baselines, thereby confirming that well-regulated signal quality, rather than mere teacher scale, governs successful exploration in OPD.