同策略蒸馏真的在蒸馏吗？从噪声教师到自我改进

Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

August 31, 2026

作者: Yi Ding, Ruqi Zhang

cs.AI

摘要

在策略蒸餾（OPD）提供了密集的詞元級監督，作為可驗證獎勵強化學習（RLVR）稀疏結果級優勢的替代方案。然而，教師對學生生成的軌跡進行評分，而這些軌跡本質上對教師而言是離策略的，因此其監督的可靠性，以及學生進步的來源，仍然不清楚。我們定量分析了OPD訓練期間的教師監督，發現存在大量噪聲，其普遍程度隨教師規模增加而上升。令人驚訝的是，學生策略對這種噪聲不敏感，無論保留還是移除噪聲監督，都能收斂到相當的性能。OPD到底有沒有在蒸餾？通過分析其收益的驅動因素，我們發現學習集中在低對數概率的詞元上，使用單一固定的負優勢即可達到與教師提供的優勢相當的性能。這表明OPD主要通過抑制低對數概率詞元來起作用，而這並不需要教師。這些發現催生了在策略自適應（OPSA），這是一種使用熵自適應負優勢的免監督方法。它對高熵位置賦予更強的學習信號，抑制尾部詞元，並在頭部詞元之間均勻重新分配概率質量。與基礎模型Qwen3-1.7B相比，OPSA在AIME24上將Avg@32提升了35.41個百分點，對應263%的相對增益，並且在所有三個基準測試上將Pass@32提升了超過一倍。它在AIME24上的Avg@32還比OPD高出16.77個百分點。跨模型家族和多項任務的大量實驗與分析進一步證明了其有效性和泛化能力。

English

On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student's improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacher-provided ones. This suggests that OPD works largely by suppressing low log-probability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens. Compared with the base Qwen3-1.7B, OPSA improves Avg@32 by 35.41 points on AIME24, corresponding to a 263\% relative gain, and more than doubles Pass@32 across all three benchmarks. It also outperforms OPD by 16.77 points in Avg@32 on AIME24. Extensive experiments and analyses across model families and tasks further demonstrate its effectiveness and generalizability.