通过直接在线策略蒸馏实现的弱到强泛化

Weak-to-Strong Generalization via Direct On-Policy Distillation

July 8, 2026

作者: Shiyuan Feng, Huan-ang Gao, Haohan Chi, Hanlin Wu, Zhilong Zhang, Zheng Jiang, Bingxiang He, Wei-Ying Ma, Ya-Qin Zhang, Hao Zhou

cs.AI

摘要

基于可验证奖励的强化学习（RLVR）是改进语言模型推理能力的一种强大方法，但每次应用于新的强模型时成本高昂，因为目标模型在训练过程中需要生成大量轨迹采样。随着模型规模扩大，后训练本身成为了一个瓶颈。我们研究了一种“弱到强”的替代方案：在轨迹采样成本更低的较小模型上运行强化学习，然后将该强化学习的结果复用来改进更强的目标模型。直接蒸馏强化学习后的弱教师模型是不够的，因为教师模型的最终策略混合了强化学习带来的有用增益与小模型的固有局限。我们提出了直接在线策略蒸馏（Direct-OPD），该方法转移的是教师模型因强化学习引起的策略偏移。Direct-OPD将强化学习后的教师与其强化学习前的参考模型进行比较，并将其对数比值作为针对学生模型的密集隐式奖励。简单来说，这一检查点对告诉我们强化学习使弱模型更倾向于或更不倾向于采取哪些动作，而Direct-OPD在更强学生模型自身的在线策略状态上应用这一信号。这种方法直接复用了弱模型强化学习的监督信号，而无需在目标模型上运行稀疏奖励强化学习。实验表明，Direct-OPD能够一致地利用弱教师模型来改进更强的目标模型；值得注意的是，它使Qwen3-1.7B在AIME 2024上的成绩从48.3%提升至58.3%，仅需在8块A100 GPU上训练4小时。该方法优于步数匹配的直接强化学习，并且能够实现多个策略偏移的序列组合。我们的结果表明，强化学习的结果可以跨模型规模作为隐式奖励信号复用，而不仅仅是作为最终模型来模仿。

English

Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model's RL supervision signal without running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.