SAF-OPD：面向在线策略蒸馏的稳定优势融合

摘要

具有可验证奖励的强化学习（RLVR）将单一的响应级奖励广播给每个词元，而同策略蒸馏（OPD）则针对每个词元与更强教师模型进行对比，以提供密集优势，但其性能上限受限于教师模型质量，并且会抑制超出该上限的探索。两者的互补性使得结合RLVR与OPD颇具前景，但我们发现，以固定系数融合这两种优势会因两种校准误差而引发熵崩溃：其一是数值量级不匹配，即词元级OPD优势可能激增至远超有界RLVR优势，从而湮没后者的信号；其二是时间维度不匹配，即持续全强度的OPD会不断将学生模型拉向教师模型，限制了超越教师模型所需的探索。为此，我们提出SAF（稳定优势融合）框架，通过一个轻量级、四阶段流水线仅对OPD优势进行处理，即可同时解决上述两个问题：采用“先稀疏化后压缩”机制进行数值量级控制，并搭配“先预热后退火”机制进行时间维度控制，其中每个阶段均可独立开关，且额外开销可忽略不计。我们以GRPO实例化RLVR，在Qwen3-1.7B/4B/8B上跨七个数学推理与代码生成基准对SAF进行评估：SAF避免了熵崩溃，并且在所有六种“模型—领域”设定中均一致优于固定系数GRPO+OPD融合方法，综合得分提升0.51%—2.70%，同时训练过程更加稳定。

English

Reinforcement learning with verifiable rewards (RLVR) broadcasts a single response-level reward to every token, while on-policy distillation (OPD) scores each token against a stronger teacher for a dense advantage but caps performance at teacher quality and discourages exploration beyond it. Their complementarity makes combining RLVR and OPD promising, but we find that fusing the two advantages with a fixed coefficient triggers entropy collapse from two miscalibrations: a magnitude mismatch, where token-level OPD advantages can spike far beyond the bounded RLVR advantage and erase its signal, and a temporal mismatch, where sustained full-strength OPD keeps pulling the student toward the teacher and limits exploration needed to surpass it. We propose SAF, a Stable Advantage Fusion framework that resolves both issues via a lightweight, four-stage pipeline applied only to the OPD advantage: a sparsify-then-compress mechanism for magnitude control paired with a warm-up-then-anneal mechanism for temporal control, with each stage independently switchable and adding negligible overhead. Instantiating RLVR with GRPO, we evaluate SAF across seven mathematical reasoning and code generation benchmarks with Qwen3-1.7B/4B/8B: SAF avoids entropy collapse and consistently outperforms fixed-coefficient GRPO+OPD fusion, improving the aggregate score by 0.51-2.70% across all six model-domain settings while achieving more stable training.