基於直接同策略蒸餾的弱至強泛化

摘要

帶可驗證獎勵的強化學習（RLVR）是提升語言模型推理能力的有效方法，但每當訓練更強的新模型時，目標模型必須在訓練過程中生成大量交互軌跡（rollouts），使得這個過程成本高昂。隨著模型規模擴大，後續訓練（post-training）本身也成為瓶頸。我們研究了一種「由弱至強」（weak-to-strong）的替代方案：先在較小模型上執行強化學習（因其交互軌跡成本較低），再將該強化學習過程中學到的成果加以重複利用，以提升更強的目標模型。直接將強化學習後的弱教師模型進行知識蒸餾並不足夠，因為教師模型的最終策略混合了強化學習帶來的有效增益與較小模型自身的侷限性。為此我們提出「直接線上策略蒸餾」（Direct On-Policy Distillation, Direct-OPD），該方法傳遞的是教師模型經由強化學習所產生的策略偏移，而非最終策略本身。Direct-OPD 將強化學習前後的教師模型進行比對，並將其對數比值（log-ratio）視為提供給學生模型的密集隱含獎勵。簡而言之，這個檢查點配對告訴我們強化學習使得較弱模型更傾向於或更不傾向於採取哪些動作，而 Direct-OPD 則將此信號應用於更強學生的在線策略狀態上。如此一來，我們直接重複利用了弱模型強化學習過程中的監督信號，而無需在目標模型上執行稀疏獎勵的強化學習。實驗顯示，Direct-OPD 能持續利用較弱教師模型來提升更強目標模型；值得關注的是，它僅需在 8 塊 A100 GPU 上訓練 4 小時，便將 Qwen3-1.7B 在 AIME 2024 上的成績從 48.3% 提升至 58.3%。該方法不僅勝過按步驟匹配的直接強化學習，還能實現多個策略偏移的序列組合。我們的研究結果表明，強化學習的成果可以透過隱含獎勵信號的形式跨模型規模重複使用，而不僅僅是做為最終模型來模仿。

English

Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model's RL supervision signal without running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 58.3% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.

基於直接同策略蒸餾的弱至強泛化

Weak-to-Strong Generalization via Direct On-Policy Distillation

摘要

Support