SPOT：用於在策略蒸餾的稀疏探測與結果校準

摘要

同策略蒸餾（OPD）透過對學生生成的軌跡提供密集的教師監督，但標準的反向KL訓練可能對其他合理的延續賦予不足的機率。僅靠教師熵無法揭示不確定性是集中在少數幾個合理的下一個詞元上，還是分散在長長的機率尾部，也無法判斷學生是否已經充分表徵了這些候選項。此外，局部的教師機率可能無法預測下游任務的成功與否。我們引入了稀疏探測與結果校正目標OPD（SPOT），透過「獲取—探索—利用」的程序，同時解決兩個相互關聯的決策：探測何處以及蒸餾什麼。在獲取階段，一個位置層級的得分結合了正規化的教師熵、小型前k候選集合所捕獲的機率質量，以及學生—教師之間的不匹配，用以分配有限的探測預算。在探索階段，SPOT透過驗證器評分的學生延續來評估教師提出的候選項。在利用階段，這些結果產生一個閉式的KL正則化目標，該目標偏好具有較佳下游結果的候選項，同時保持錨定於教師分佈。跨多個學生模型與推理基準的大量實驗證明了SPOT在提升推理表現、同時平衡解決方案品質與覆蓋度方面的有效性。

English

On-policy distillation (OPD) provides dense teacher supervision on student-generated trajectories, but standard reverse-KL training can assign insufficient probability to other plausible continuations. Teacher entropy alone does not reveal whether uncertainty is concentrated among a few plausible next tokens or dispersed over a long probability tail, nor whether the student already represents those candidates well. Moreover, local teacher probabilities may not predict downstream success. We introduce Sparse Probing and Outcome-calibrated Targets OPD (SPOT), which addresses two coupled decisions, where to probe and what to distill, through an acquisition--exploration--exploitation procedure. During acquisition, a position-level score combines normalized teacher entropy, the probability mass captured by a small top-k candidate set, and student--teacher mismatch to allocate a limited probing budget. During exploration, SPOT evaluates teacher-proposed candidates through verifier-scored student continuations. During exploitation, these outcomes produce a closed-form, KL-regularized target that favors candidates with better downstream outcomes while remaining anchored to the teacher distribution. Extensive experiments across multiple student models and reasoning benchmarks demonstrate the effectiveness of SPOT in improving reasoning performance while balancing solution quality and coverage.