SAF-OPD：用於在策略蒸餾的穩定優勢融合

摘要

具可驗證獎勵的強化學習（RLVR）將單一回應層級的獎勵廣播至每個詞元；而在策略蒸餾（OPD）則以更強的教師模型逐詞元評分，以提供稠密優勢，但其性能上限受限於教師品質，且會抑制超越教師品質的探索。兩者的互補性使得結合 RLVR 與 OPD 極具前景，但我們發現，以固定係數融合這兩種優勢會因兩類校準失誤而引發熵崩潰：一是量級失配，詞元級 OPD 優勢可能飆升至遠超有界的 RLVR 優勢，從而抹除其訊號；二是時間失配，持續全強度的 OPD 不斷將學生模型拉向教師模型，限制了超越教師所需的探索。我們提出 SAF，一個穩定優勢融合框架，透過僅應用於 OPD 優勢的輕量級四階段管線解決上述兩個問題：以「先稀疏化再壓縮」機制進行量級控制，並搭配「先預熱再退火」機制進行時間控制；每個階段均可獨立開關，且額外開銷可忽略不計。我們以 GRPO 實例化 RLVR，在七個數學推理與程式碼生成基準上、使用 Qwen3-1.7B/4B/8B 評估 SAF：SAF 避免了熵崩潰，且在所有六個模型-領域設定中持續優於固定係數的 GRPO+OPD 融合，將綜合分數提升 0.51% 至 2.70%，同時實現更穩定的訓練。

English

Reinforcement learning with verifiable rewards (RLVR) broadcasts a single response-level reward to every token, while on-policy distillation (OPD) scores each token against a stronger teacher for a dense advantage but caps performance at teacher quality and discourages exploration beyond it. Their complementarity makes combining RLVR and OPD promising, but we find that fusing the two advantages with a fixed coefficient triggers entropy collapse from two miscalibrations: a magnitude mismatch, where token-level OPD advantages can spike far beyond the bounded RLVR advantage and erase its signal, and a temporal mismatch, where sustained full-strength OPD keeps pulling the student toward the teacher and limits exploration needed to surpass it. We propose SAF, a Stable Advantage Fusion framework that resolves both issues via a lightweight, four-stage pipeline applied only to the OPD advantage: a sparsify-then-compress mechanism for magnitude control paired with a warm-up-then-anneal mechanism for temporal control, with each stage independently switchable and adding negligible overhead. Instantiating RLVR with GRPO, we evaluate SAF across seven mathematical reasoning and code generation benchmarks with Qwen3-1.7B/4B/8B: SAF avoids entropy collapse and consistently outperforms fixed-coefficient GRPO+OPD fusion, improving the aggregate score by 0.51-2.70% across all six model-domain settings while achieving more stable training.