更密集≠更好：持續後訓練中同策略自我蒸餾的局限性

摘要

持續後訓練使基礎模型能夠學習新知識，同時保留既有能力。近期研究指出，在線策略學習可減緩遺忘，其中在線策略自蒸餾特別受到關注。本研究透過自蒸餾策略最佳化（SDPO）重新檢視此樂觀看法。實驗顯示，當教師訊號穩定且對齊良好時，SDPO能加速領域內特化，但難以泛化至分佈外情境。在持續後訓練中，SDPO表現出更強的遺忘現象，甚至可能崩潰，而GRPO等基於在線策略的強化學習方法則以更保守的方式適應，並較好地保留先前能力。進一步分析發現，密集自蒸餾在參數空間與回應空間皆導致較大的偏移，並可能透過自我強化的師生迴圈放大高頻格式偽影。這些發現表明，僅靠線上策略資料不足以實現持續學習。密集自蒸餾雖能在教師目標穩定且詞元級監督可靠時加速特化，但不應視為持續後訓練的預設穩定器。我們的程式碼已開源於 https://github.com/Moenupa/SDPO-CL。

English

Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as a particularly attractive approach. In this work, we revisit this optimistic view through self-distillation policy optimization (SDPO). Our experiments show that SDPO can accelerate in-domain specialization when teacher signals are stable and well aligned, but it struggles to generalize to out-of-distribution scenarios. In continual post-training, SDPO exhibits stronger forgetting and can even collapse, whereas on-policy reinforcement learning methods such as GRPO adapt more conservatively and better preserve prior capabilities. Further analyses reveal that denser self-distillation induces larger drift in both parameter space and response space, and can amplify high-frequency formatting artifacts through a self-reinforcing teacher--student loop. These findings suggest that on-policy data alone is insufficient for continual learning. Dense self-distillation can accelerate specialization when teacher targets are stable and token-level supervision is reliable, but it should not be treated as a default stabilizer for continual post-training. Our code is available at https://github.com/Moenupa/SDPO-CL.