在策略蒸馏真的在蒸馏吗？从噪声教师到自我提升

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

August 31, 2026

作者: Yi Ding, Ruqi Zhang

cs.AI

摘要

在线策略蒸馏（OPD）提供密集的token级监督，作为可验证奖励强化学习（RLVR）中稀疏结果级优势的替代方案。然而，教师对学生生成的轨迹进行评分，而这些轨迹本质上对其而言是离策略的，因此其监督的可靠性，以及学生改进的来源，仍不清楚。我们定量分析了OPD训练过程中的教师监督，发现存在大量噪声，且其普遍性随教师规模增大而增加。令人惊讶的是，学生策略对此类噪声不敏感，无论保留还是移除噪声监督，均能收敛到相当的性能。OPD究竟是否在进行蒸馏？通过分析其收益的驱动因素，我们发现学习集中在低对数概率的token上，且使用单一固定的负优势值即可达到与教师提供的优势值相当的性能。这表明OPD主要通过抑制低对数概率token来发挥作用，而这一过程并不需要教师。这些发现催生了在线策略自适应（OPSA）——一种利用熵自适应负优势值的无监督方法。它对高熵位置施加更强的学习信号，抑制尾部token，并在头部token之间均匀重新分配概率质量。与基础Qwen3-1.7B相比，OPSA在AIME24上将Avg@32提升了35.41分，对应263%的相对提升，并且在所有三个基准上将Pass@32提升了一倍以上。在AIME24上，其Avg@32还比OPD高出16.77分。跨模型家族和任务的大量实验与分析进一步证明了其有效性和泛化能力。

English

On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.