同策略增量蒸馏

摘要

在线策略蒸馏是强化学习中一种替代性的后训练方法，它通过从教师模型提供词元级别的监督信号，缓解了奖励模型带来的约束。尽管在线策略蒸馏已在多种场景下得到研究和应用，但其基本设计仍未被充分探索。本文提出一种新的蒸馏奖励——增量信号（delta signal），以替代直接模仿教师模型的输出分布。增量信号定义为教师模型与其在推理能力指令微调前的基础模型之间的差值。因此，它能够捕捉推理微调所引发的变化，并为迁移推理能力提供更直接的信号。通过大量实证证据，我们证明增量信号显著改进了在线策略蒸馏，并将这种新蒸馏方法命名为在线策略增量蒸馏（On-Policy Delta Distillation, OPD²）。在数学、科学和代码推理基准上的实验表明，OPD²始终优于传统在线策略蒸馏，使推理大语言模型仅需短暂的后训练即可实现强劲性能。代码将在 https://github.com/naver-ai/opd2 上公开。

English

On-policy distillation is an alternative post-training method in reinforcement learning that alleviates the constraints imposed by reward models by providing token-level supervision from a teacher model. Although on-policy distillation has been studied and applied across various settings, its fundamental design remains underexplored. In this paper, we introduce a new distillation reward, termed the delta signal, instead of directly imitating the teacher's output distribution. The delta signal is defined as the difference between the teacher model and its base model prior to instruction tuning for reasoning capability. It therefore captures the changes induced by reasoning tuning and provides a more direct signal for transferring reasoning capabilities. Using extensive empirical evidence, we show that the delta signal substantially improves on-policy distillation and refer to the new distillation method as On-Policy Delta Distillation (OPD^2). Experiments across mathematics, science, and code-reasoning benchmarks demonstrate that OPD^2 consistently outperforms conventional on-policy distillation, enabling reasoning LLMs to achieve strong performance with only a short post-training period. Code will be available at https://github.com/naver-ai/opd2