一個症狀，三個槓桿：在策略自蒸餾之批判性回顧

摘要

在策略蒸餾（on-policy distillation）會讓語言模型以自己的生成結果進行訓練，同時由教師模型逐詞元（token）評分。此方法結合了模仿學習的密集監督，以及強化學習的在策略取樣。然而，它需要第二個更大的模型來擔任教師。在策略自蒸餾（On-Policy Self-Distillation，OPSD）消除了這項成本。教師即為模型本身，但以學生在測試時無法取得的特權資訊（privileged information）作為條件，例如參考解答、計畫或環境回饋。教師並不比學生更強，只是擁有更充分的資訊。早期結果相當有前景，僅需生成一小部分詞元，即可達到與強化學習相當的準確度。然而，正是這種產生訊號的不對稱性，也使得訊號產生偏差。目前該領域由一種失敗模式主導：坍縮（collapse），亦即模型所能產生的推理路徑集合逐漸變窄。坍縮並非 OPSD 所獨有，但特權資訊會加劇此現象。本綜述將坍縮視為受三個槓桿調控的症狀：（i）訊號施加的位置，亦即詞元如何被加權；（ii）教師所見的內容，亦即特權資訊的性質；以及（iii）訊號何時改變，亦即教師的動態與引導的衰減。我們將範疇限定於數學推理，因為此方法起源於此，且其失敗模式在此有最完整的記錄。我們未報告新實驗。本文的貢獻在於結構性層面：為不同論文中以不同名稱描述的現象提供共享詞彙，並清楚劃分已定論與仍具爭議之處。

English

On-policy distillation trains a language model on its own generations while a teacher scores them token by token. It combines the dense supervision of imitation learning with the on-policy sampling of reinforcement learning. But it requires a second, larger model to act as teacher. On-Policy Self-Distillation (OPSD) removes that cost. The teacher is the model itself, conditioned on privileged information the student will not have at test time, such as a reference solution, a plan, or environment feedback. The teacher is no stronger than the student, only better informed. Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens. But the same asymmetry that produces the signal also biases it. One failure mode now dominates the field: collapse, the progressive narrowing of the set of reasoning paths the model can produce. Collapse is not specific to OPSD, though privileged information aggravates it. This review treats collapse as a symptom governed by three levers: (i) where the signal is applied, that is, how tokens are weighted; (ii) what the teacher is shown, that is, the nature of the privileged information; and (iii) when the signal changes, that is, the teacher's dynamics and the decay of guidance. We restrict our scope to mathematical reasoning, where the method originated and where its failure modes are best documented. We report no new experiments. The contribution is structural: a shared vocabulary for phenomena named differently across papers, and a clear line between what is settled and what is still disputed.