影响引导蒸馏：解决采样词元同策略蒸馏中的多样性瓶颈

摘要

基于采样词元的在线蒸馏（OPD）高效地将能力从教师模型迁移至学生模型，其利用学生模型生成的词元，且仅需采样词元对应的教师概率。然而，该方法常遭遇多样性蒸馏失效问题：学生的pass@1提升，而pass@k却停滞不前，未能继承教师的多样性。为解释该现象，我们引入一阶局部熵影响（First-Order Local Entropy Influence），这是一种带符号的一阶代理指标，可将每次更新的熵效应解耦为师生对数概率差与学生局部概率结构两部分，并从经验上将熵收缩与负影响位置相关联。基于此，我们提出影响引导的自适应在线蒸馏（Influence-Directed Adaptive On-Policy Distillation, IDA-OPD）：该方法不依赖成本高昂的全词表前向KL目标，而是保留熵增更新，同时用散度自适应优势收缩替代熵减更新，且仅使用教师对采样词元的对数概率。在面向推理的蒸馏实验表明，IDA-OPD一致性地提升pass@k，通过蒸馏继承教师的多样性，以严格更低的成本达到最强教师信息引导方法的性能，并在无需全词表教师信息的情况下广泛维持原始OPD的pass@1水平。

English

Sampled-token on-policy distillation (OPD) efficiently transfers capabilities from teacher to student using student-generated tokens, requiring teacher probabilities only for sampled tokens. Yet it frequently suffers from diversity distillation failure: the student's pass@1 improves while its pass@k plateaus, failing to inherit the teacher's diversity. To explain this, we introduce First-Order Local Entropy Influence, a signed first-order proxy that decouples each update's entropy effect into the teacher--student log-probability gap and the student's local probability structure, and empirically links entropy contraction to negative-influence positions. Motivated by this, we propose Influence-Directed Adaptive On-Policy Distillation (IDA-OPD): rather than relying on costly full-vocabulary Forward-KL objectives, it preserves entropy-expanding updates while replacing entropy-contracting ones with divergence-adaptive advantage shrinkage, using only the teacher's sampled-token log-probability. Experiments on reasoning-oriented distillation show IDA-OPD consistently improves pass@k, inheriting the teacher's diversity through distillation, matches the strongest teacher-informed methods at strictly lower cost, and broadly maintains vanilla OPD's pass@1, all without full-vocabulary teacher information.