同策略增量蒸馏

摘要

同策略蒸餾是強化學習中的一種替代性後訓練方法，透過教師模型提供的詞元層級監督，來減輕獎勵模型所施加的限制。儘管同策略蒸餾已在多種場景中被研究與應用，其基本設計仍未充分探索。本文提出一種新的蒸餾獎勵，稱為增量訊號，用以取代直接模仿教師的輸出分佈。增量訊號定義為教師模型與其進行推理能力指令微調前的基礎模型之間的差異。因此，它捕捉了推理微調所帶來的變化，並提供更直接的訊號以轉移推理能力。透過大量實證證據，我們證明增量訊號能顯著提升同策略蒸餾，並將此新蒸餾方法命名為同策略增量蒸餾（OPD^2）。在數學、科學與程式推理基準測試上的實驗結果顯示，OPD^2 一致優於傳統的同策略蒸餾，使推理大型語言模型僅需短暫的後訓練即可達成優異表現。程式碼將於 https://github.com/naver-ai/opd2 提供。

English

On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD^2). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD^2 consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at https://github.com/naver-ai/opd2