更密集 ≠ 更好：同策略自蒸馏在持续后训练中的局限性

摘要

持续后训练使基础模型能够获取新知识，同时保持已有能力。近期研究表明，在线策略学习可缓解遗忘，其中在线策略自蒸馏成为一种颇具吸引力的方法。本文通过自蒸馏策略优化（SDPO）重新审视了这一乐观观点。实验结果表明，当教师信号稳定且对齐良好时，SDPO能加速领域内特化，但在处理分布外场景时难以泛化。在持续后训练中，SDPO表现出更强的遗忘现象，甚至可能崩溃，而GRPO等在线策略强化学习方法则能以更保守的方式适应，更好地保留先前能力。进一步分析发现，更密集的自蒸馏会在参数空间和响应空间中引发更大漂移，并通过自我强化的师生循环放大高频格式伪影。这些发现表明，仅凭在线策略数据不足以实现持续学习。密集自蒸馏在教师目标稳定且令牌级监督可靠时能加速特化，但不应被视为持续后训练的默认稳定器。我们的代码已开源，见 https://github.com/Moenupa/SDPO-CL。

English

Continual post-training enables foundation models to acquire new knowledge while preserving existing capabilities. Recent work suggests that on-policy learning can mitigate forgetting, with on-policy self-distillation emerging as a particularly attractive approach. In this work, we revisit this optimistic view through self-distillation policy optimization (SDPO). Our experiments show that SDPO can accelerate in-domain specialization when teacher signals are stable and well aligned, but it struggles to generalize to out-of-distribution scenarios. In continual post-training, SDPO exhibits stronger forgetting and can even collapse, whereas on-policy reinforcement learning methods such as GRPO adapt more conservatively and better preserve prior capabilities. Further analyses reveal that denser self-distillation induces larger drift in both parameter space and response space, and can amplify high-frequency formatting artifacts through a self-reinforcing teacher--student loop. These findings suggest that on-policy data alone is insufficient for continual learning. Dense self-distillation can accelerate specialization when teacher targets are stable and token-level supervision is reliable, but it should not be treated as a default stabilizer for continual post-training. Our code is available at https://github.com/Moenupa/SDPO-CL.