先驗證，再蒸餾：針對在策略蒸餾的提示層級教師門控

摘要

在策略蒸餾（On-policy Distillation, OPD）透過在學生模型自身生成的軌跡上，由凍結的教師模型提供密集的 token 層級監督，加速後訓練。基本 OPD 將此監督均勻套用於所有提示，而未檢查教師模型對每個提示是否可靠。由於反向 KL 散度具有尋模態（mode-seeking）特性，一個自信但錯誤的教師可能引發強烈且具誤導性的更新。分布層面的代理指標（如熵，或教師與學生之間的似然一致性）能衡量不確定性或一致程度，但無法直接驗證結果的正確性。我們提出教師門控在策略蒸餾（Teacher-Gated On-Policy Distillation, TGOPD），其原則為：在允許密集監督之前，必須先在提示層級驗證教師的可靠性。TGOPD 從一小組由驗證器評分的教師探測樣本中估計可靠性，並在可靠性檢查通過時，將每個提示專門導向密集 OPD；否則，導向基於驗證器的 GRPO。在數學、程式碼與指令跟隨任務上，無論是 4B 還是 35B 的學生模型，TGOPD 在全部六種單一領域設定中皆優於基本 OPD，並在多領域訓練中於兩種模型規模下均達到更高的七項基準平均分數。此外，TGOPD 利用原本閒置的教師算力進行可靠性估計，因而降低了非同步 OPD 中教師端的算力浪費；在所量測的 4B 單一領域執行中，教師節點 GPU 利用率從 9.8% 提升至 78.9%。

English

On-policy distillation (OPD) accelerates post-training by providing dense token-level supervision from a frozen teacher on the student's own rollouts. Vanilla OPD applies this supervision uniformly across prompts, without checking whether the teacher is reliable for each prompt. Because reverse KL is mode-seeking, a confidently wrong teacher can induce a strong yet misleading update. Distributional proxies, such as entropy or teacher-student likelihood agreement, measure uncertainty or agreement but do not directly verify outcome correctness. We introduce Teacher-Gated On-Policy Distillation (TGOPD), built on the principle that teacher reliability should be verified at the prompt level before dense supervision is admitted. TGOPD estimates reliability from a small set of verifier-scored teacher probes and routes each prompt exclusively to dense OPD when the reliability check passes or to verifier-grounded GRPO otherwise. Across 4B and 35B students in mathematics, code, and instruction following, TGOPD outperforms Vanilla OPD in all six single-domain settings and achieves higher seven-benchmark averages at both scales under multi-domain training. By using otherwise-idle teacher capacity for reliability estimation, TGOPD also reduces teacher-side compute waste in asynchronous OPD, increasing teacher-node GPU utilization from 9.8% to 78.9% in the measured 4B single-domain run.