大语言模型后训练的蒸馏强化学习

摘要

大语言模型（LLM）后训练对于提升推理能力、适应性和对齐至关重要。现有方法主要遵循两种范式：强化学习（RL）和同策略蒸馏（OPD）。然而，强化学习依赖于粗粒度的结果监督，导致信用分配困难，且获取新知识的能力有限。而OPD通过KL散度无条件地匹配教师模型的对数概率，这引发了一个困境：相似的教师模型无法提供新知识，而差异显著的教师模型往往产生无效指导，这极大地限制了OPD仅适用于同族蒸馏。我们提出蒸馏强化学习（Distilled RL），将教师监督融入RL目标中，以提供细粒度指导、有选择地迁移新知识并避免无条件模仿。Distilled RL包含三个组件：带裁剪的逆重要性采样、负样本重置和序列级几何归一化。通过一个简洁且可解释的案例研究，我们证明了Distilled RL能够有效将教师模型中先前不可获取的知识迁移到学生模型。在跨同族和跨族蒸馏设置的大量实验表明，Distilled RL在pass@1和pass@k指标上均显著优于标准RL和OPD。我们的代码开源在https://github.com/597358816/Distilled-RL。

English

Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new knowledge, while substantially different teachers often yield ineffective guidance, largely restricting OPD to within-family distillation. We propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision into the RL objective to provide fine-grained guidance, selectively transfer new knowledge and avoid unconditional imitation. Distilled RL contains three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization. Through a concise and interpretable case study, we demonstrate that Distilled RL can effectively transfer previously unavailable knowledge from a teacher model to a student model. Extensive experiments across both within-family and cross-family distillation settings show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k. Our code is available at https://github.com/597358816/Distilled-RL.