从失败处蒸馏：从自适应教师指导中恢复负向RL组的学习信号

摘要

基于可验证奖励的强化学习（RLVR）已成为大语言模型（LLM）后训练的标准范式。尽管组相对策略优化（GRPO）被广泛采用，但它存在奖励信号稀疏的问题，并且当组内所有响应获得相同奖励时，梯度会完全消失。在策略蒸馏（OPD）通过提供来自教师模型的密集的、词元级别的监督信号，提供了一种自然的补救措施。然而，将GRPO与OPD朴素相结合会导致性能下降，其原因有三：并非所有样本都能从蒸馏中受益；过快拟合教师模型会削弱强化学习的探索能力；OPD的优势是不对称的，会抑制大多数词元。针对这些挑战，我们提出了RSTG（通过自适应教师指导恢复学习信号），该方法在最为关键之处选择性地、精准地应用蒸馏。在样本层面，OPD仅被限制应用于零方差负奖励提示，且每个样本按教师模型的置信度分数加权。在词元层面，蒸馏仅针对学生模型熵较高或教师-学生差异较大的词元。我们进一步使用教师模型生成的正确轨迹进行SFT来增强训练，在强化学习无法产生梯度的地方注入正向梯度信号。实验表明，RSTG在数学任务上比朴素GRPO+OPD显著提升4.02%，在代码任务上提升3.05%。

English

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models (LLMs). While Group Relative Policy Optimization (GRPO) is widely adopted, it suffers from sparse reward signals and loses gradients entirely when all responses within a group receive identical rewards. On-policy distillation (OPD) offers a natural remedy by providing dense, token-level supervision from a teacher model. However, naively combining GRPO with OPD leads to degraded performance, due to three underlying causes: not all samples benefit from distillation; fitting too quickly to the teacher undermines the exploratory capacity of RL; and OPD's advantages are asymmetric, suppressing most tokens. To address these challenges, we propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most. At the sample level, OPD is restricted to negative zero-variance prompts with each sample weighted by the teacher's confidence score. At the token level, distillation targets only tokens with high student entropy or large teacher-student divergence. We further augment training with SFT on correct trajectories generated by the teacher model, injecting positive gradient signals where RL yields none. Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and +3.05% on code.