当教师误导时：虚假信号感知的在线策略蒸馏

摘要

同策略蒸馏（On-Policy Distillation, OPD）通过使用密集的token级教师信号来监督学生模型采样的轨迹，从而迁移教师能力。近期的选择性OPD方法通过优先处理高置信度、信息丰富或可学习的信号来改进该过程。然而，这些假设忽略了一个语言模型的根本性失效模式：其token级判断可能由与输入无关的语言先验、格式惯例或程式化推理模板所驱动，而非任务特定证据。我们将这种与优化相关但输入依据较弱的监督称为OPD中的伪信号（spurious signals）；它们可能产生较大的梯度，却对任务改进方向贡献甚微。为缓解这一问题，我们提出SA-OPD，一种伪信号感知的同策略蒸馏框架，基于输入依据性和优化影响来识别并过滤具有误导性的token级监督。SA-OPD引入一个轻量级的输入依据性代理，用于估计token级蒸馏信号是否真正依赖于输入；随后仅过滤同时呈现低输入依据性和极端蒸馏差异的token，从而去除高影响的伪更新，实现细粒度的OPD优化。在大型语言模型（LLM）和视觉语言模型（VLM）设置上的大量实验表明，SA-OPD始终优于Vanilla OPD和具有竞争力的选择性方法。这些结果确立了输入依据性作为OPD监督选择的一个关键维度，并为缓解伪更新提供了一种简单有效的策略。

English

On-Policy distillation (OPD) transfers teacher capabilities by supervising student-sampled trajectories with dense token-level teacher signals. Recent selective OPD methods improve this process by prioritizing signals that are confident, informative, or learnable. However, the assumptions overlook a fundamental failure mode of language models: their token-level judgments can be driven by input-agnostic language priors, formatting conventions, or stereotyped reasoning templates rather than task-specific evidence. We refer to such optimization-relevant but weakly input-grounded supervision as spurious signals in OPD, which may produce large gradients while contributing little task-improving direction. To mitigate this issue, we propose SA-OPD, a Spurious-Signal-Aware On-Policy Distillation framework that identifies and filters misleading token-level supervision based on input-groundedness and optimization impact. SA-OPD introduces a lightweight input-groundedness proxy estimating whether a token-level distillation signal truly depends on the input. It then filters only tokens that simultaneously exhibit low input-groundedness and extreme distillation divergence, thereby removing high-impact spurious updates and achieving fine-grained OPD optimization. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that SA-OPD consistently outperforms Vanilla OPD and competitive selective methods. These results establish input-groundedness as a key dimension for OPD supervision selection and offer a simple, effective strategy for mitigating spurious updates.