DAPD：雙錨定策略蒸餾

摘要

在策略（自）蒸餾（OPSD）日益被採用於語言模型的後訓練。它利用特權資訊強化教師模型，但可能引發特權幻覺：學生模型學習到依賴特權的行為，這些行為無法從推理時的上下文重現，卻表現得彷彿訓練時的特權資訊仍然可用，最終導致效能下降。在本文中，我們將推理時特權教師與學生之間的資訊不對稱，識別為 OPSD 中此失敗的根本原因。為了解決這種不對稱，我們提出了雙錨定策略蒸餾（Dual-Anchored Policy Distillation, DAPD），這是一個具有兩層錨定的統一框架。雙路徑錨定（Dual-Path Anchoring, DPA）引入了一個自條件橋樑，並沿兩條資訊匹配的路徑對齊參考與展開行為，防止依賴特權的行為被轉移到推理時的學生模型。雙來源錨定（Dual-Source Anchoring, DSA）在參考到展開以及展開到參考兩個方向上應用這些路徑，在減少對特權參考引導依賴的同時，保留正確性監督。大量實驗表明，DAPD 顯著緩解了特權幻覺，在 Qwen3-4B 上的各項任務平均比 OPSD 高出 +2.00 分。值得注意的是，其收益在不同規模下持續存在，在 4B 規模達到 +2.69，在 32B 規模達到 +2.78。

English

On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.