於失敗處蒸餾：從自適應教師引導中恢復負面 RL 組群的學習信號

摘要

可驗證獎勵的強化學習（RLVR）已成為大型語言模型（LLMs）後續訓練的標準範式。儘管群體相對策略最佳化（GRPO）被廣泛採用，但其存在獎勵訊號稀疏的問題，且當群組內所有回應獲得相同獎勵時，梯度會完全消失。在線策略蒸餾（OPD）透過提供來自教師模型的密集、token層級監督，提供了一種自然的解決方案。然而，單純將GRPO與OPD結合會導致效能下降，其原因有三：並非所有樣本都能從蒸餾中受益；過快貼合教師模型會削弱強化學習的探索能力；且OPD的優勢具有不對稱性，會抑制大部分token。為了解決這些挑戰，我們提出RSTG（透過自適應教師引導恢復學習訊號），該方法選擇性地且精確地在最關鍵之處應用蒸餾。在樣本層級，OPD僅限應用於負向零變異提示，且每個樣本均以教師模型的信心分數加權。在token層級，蒸餾僅針對具有高學生熵或較大教師-學生分歧度的token。我們進一步以教師模型生成的正確軌跡進行監督式微調（SFT）來增強訓練，在強化學習無法產生梯度的情況下注入正向梯度訊號。實驗表明，RSTG在數學任務上比單純的GRPO+OPD顯著提升+4.02%，在程式碼任務上提升+3.05%。

English

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models (LLMs). While Group Relative Policy Optimization (GRPO) is widely adopted, it suffers from sparse reward signals and loses gradients entirely when all responses within a group receive identical rewards. On-policy distillation (OPD) offers a natural remedy by providing dense, token-level supervision from a teacher model. However, naively combining GRPO with OPD leads to degraded performance, due to three underlying causes: not all samples benefit from distillation; fitting too quickly to the teacher undermines the exploratory capacity of RL; and OPD's advantages are asymmetric, suppressing most tokens. To address these challenges, we propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most. At the sample level, OPD is restricted to negative zero-variance prompts with each sample weighted by the teacher's confidence score. At the token level, distillation targets only tokens with high student entropy or large teacher-student divergence. We further augment training with SFT on correct trajectories generated by the teacher model, injecting positive gradient signals where RL yields none. Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and +3.05% on code.