基於可驗證獎勵的同策略蒸餾

摘要

基於可驗證獎勵的強化學習（RLVR）與同策略蒸餾（OPD）已成為大型語言模型後訓練的兩種廣泛採用範式。然而，RLVR 存在任務層級回饋稀疏的問題，而 OPD 雖提供密集的 token 層級引導，卻忽略軌跡的正確性，導致其效能受限於教師模型。結合兩者是一個具前景的方向：OPD 提供密集的監督訊號，而 RLVR 提供任務層級的正確性。然而，現有的整合方式常依賴加權組合或啟發式切換，引入額外的超參數與取捨。我們提出「具可驗證獎勵的同策略蒸餾」（OPDVR），這是一種簡單而有效的方法，能在不增加任何超參數的情況下無縫結合 OPD 與 RLVR。我們首先根據軌跡正確性重新建構採樣 token OPD 的隱含獎勵，接著套用 ReLU 門控機制，確保正確軌跡獲得非負獎勵，錯誤軌跡獲得非正獎勵——藉此將蒸餾訊號與任務成功對齊，同時保留教師模型的分佈引導。此外，我們的修改將採樣 token OPD 轉化為正式的 RLVR 方法，使其能輕鬆與任何策略梯度演算法（如 GRPO）結合。在六個推理基準上的實驗顯示，OPDVR 持續優於標準 OPD。我們的程式碼可在 https://github.com/LeapLabTHU/OPDVR 取得。

English

Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD) have become two widely adopted paradigms for post-training large language models. However, RLVR suffers from sparse task-level feedback, while OPD provides dense token-level guidance but ignores trajectory correctness, limiting its performance to that of the teacher. Combining them is a promising direction: OPD supplies dense supervisory signals, while RLVR provides task-level correctness. Nevertheless, existing integrations often rely on weighted combination or heuristic switching, introducing extra hyperparameters and trade-offs. We propose On-policy Distillation with Verifiable Reward (OPDVR), a simple yet effective method that seamlessly combines OPD and RLVR without adding any hyperparameters. We first reformulate the implicit reward of sampled-token OPD based on trajectory correctness, then apply a ReLU gating mechanism to ensure that correct trajectories receive non-negative rewards and incorrect ones receive non-positive rewards---thereby aligning the distillation signal with task success while preserving the teacher's distributional guidance. Furthermore, our modification transforms sampled-token OPD into a proper RLVR method, making it readily combinable with any policy gradient algorithm, such as GRPO. Experiments on six reasoning benchmarks show that OPDVR consistently outperforms standard OPD. Our code is available at https://github.com/LeapLabTHU/OPDVR.