揭秘在策略蒸馏：角色、病态与调控

摘要

同策略蒸馏已成为大语言模型后训练中的关键范式，但其训练动态仍缺乏深入理解。我们开展系统性研究，审视同策略蒸馏的作用、病态特征与调控机制。首先明确同策略蒸馏作为探索催化剂的功能：通过密集的令牌级引导，在不突破能力上限的前提下，将学生模型导向正确推理路径。通过实验证实，提示多样性比单问题采样数量更重要，且关键在于同策略蒸馏的有效性完全依赖于引导信号的质量。这种依赖性暴露出两种破坏探索的病态特征：当师生分布差距过大导致引导信号与任务正确性错位时，会产生师生失配现象，使探索方向适得其反；当聚合的令牌级目标产生长度相关捷径时，会引发长度利用问题，学生模型通过响应截断或冗余填充操纵奖励景观，从而探索退化长度模式而非推理策略。为遏制这些病态特征，我们研究轻量级信号调控机制：优势裁剪与对数尺度压缩，确保探索过程由可信信号引导。跨七个基准的实验表明，这些调控能缓解长度利用问题，实现有效蒸馏，稳定超越同策略蒸馏变体与强化学习验证推理基准，证实良好的信号质量（而非单纯的教师模型规模）才是同策略蒸馏中成功探索的决定性因素。

English

On-policy distillation (OPD) has become a key paradigm in LLM post-training, yet its training dynamics remain poorly understood. We present a systematic study examining the role, pathologies, and regulations of OPD. We first clarify the role of OPD as an exploration catalyst: it steers the student toward correct reasoning paths via dense token-level guidance, without expanding capability ceiling. We confirm this by showing that prompt diversity matters more than per-problem sampling numbers, and critically, that the effectiveness of OPD hinges entirely on the quality of its guiding signal. This dependency exposes two pathologies that derail exploration. The Student-Teacher Mismatch occurs when a large teacher-student distributional gap causes the guiding signal to misalign with task correctness, steering exploration in counterproductive directions. Length Exploitation arises when the aggregated token-level objective creates length-dependent shortcuts, allowing the student to game the reward landscape through response truncation or redundant padding, exploring degenerate length modes rather than reasoning strategies. To tame these pathologies, we investigate lightweight signal regulations: advantage clipping and log-scale compression, ensuring exploration is guided by faithful signals. Experiments across seven benchmarks demonstrate that these regulations alleviate length exploitation and enable effective distillation, stably surpassing OPD variants and RLVR baselines, thereby confirming that well-regulated signal quality, rather than mere teacher scale, governs successful exploration in OPD.