DAPD：双锚定策略蒸馏

摘要

在策略（自）蒸馏（OPSD）正越来越多地被应用于语言模型后训练。它通过特权信息强化教师模型，但可能引发特权幻觉：学生模型学习到依赖特权信息的行为，而这些行为在推理时无法从上下文中复现，然而学生模型却表现得仿佛训练时的特权信息仍然可用，最终导致性能下降。在本文中，我们指出特权教师与推理阶段学生模型之间的信息不对称是 OPSD 中这一失败的根源。为解决这一不对称问题，我们提出双锚定策略蒸馏（DAPD），一种具有两层锚定的统一框架。双路径锚定（DPA）引入自条件桥接，并沿两条信息匹配路径对齐参考行为与采样行为，防止依赖特权信息的行为被迁移到推理时的学生模型。双源锚定（DSA）在参考到采样和采样到参考两个方向上应用这些路径，减少对特权参考指导的依赖，同时保留正确性监督。大量实验表明，DAPD 显著缓解了特权幻觉，在 Qwen3-4B 上各项任务平均超越 OPSD +2.00 分。值得注意的是，其增益在不同规模上持续存在，在 4B 规模达到 +2.69，在 32B 规模达到 +2.78。

English

On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.