基于可验证奖励的在线策略蒸馏

摘要

基于可验证奖励的强化学习（RLVR）与同策略蒸馏（OPD）已成为大语言模型后训练中两种广泛采用的范式。然而，RLVR面临稀疏的任务级反馈，而OPD提供稠密的词元级指导，却忽略了轨迹正确性，使其性能局限于教师模型。将二者结合是一个有前景的方向：OPD提供稠密监督信号，RLVR提供任务级正确性。尽管如此，现有集成方法往往依赖加权组合或启发式切换，引入了额外超参数与权衡。我们提出OPDVR（基于可验证奖励的同策略蒸馏），一种简单却有效的方法，无需添加任何超参数即可无缝结合OPD与RLVR。我们首先基于轨迹正确性重新表述采样词元同策略蒸馏的隐式奖励，然后应用ReLU门控机制，确保正确轨迹获得非负奖励，错误轨迹获得非正奖励——从而在保留教师模型分布指导的同时，使蒸馏信号与任务成功对齐。此外，我们的修改将采样词元同策略蒸馏转化为一种真正的RLVR方法，使其能够与任意策略梯度算法（如GRPO）直接结合。在六个推理基准上的实验表明，OPDVR持续优于标准OPD。我们的代码已开源：https://github.com/LeapLabTHU/OPDVR。

English

Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training large language models. However, RLVR suffers from sparse task-level feedback, while OPD provides dense token-level guidance but ignores trajectory correctness, limiting its performance to that of the teacher. Combining them is a promising direction: OPD supplies dense supervisory signals, while RLVR provides task-level correctness. Nevertheless, existing integrations often rely on weighted combination or heuristic switching, introducing extra hyperparameters and trade-offs. We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR without adding any hyperparameters. We first reformulate the implicit reward of sampled-token OPD based on trajectory correctness, then apply a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards---thereby aligning the distillation signal with task success while preserving the teacher's distributional guidance. Furthermore, our modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO. Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD. Our code is available at https://github.com/LeapLabTHU/OPDVR.