大語言模型後訓練的蒸餾強化學習

摘要

大型語言模型（LLM）的後期訓練對於提升推理能力、適應性與對齊至關重要。現有方法主要遵循兩種範式：強化學習（RL）與在策略蒸餾（OPD）。然而，RL依賴於粗粒度的結果監督，導致信用分配困難且獲取新知識的能力有限。另一方面，OPD透過KL散度無條件地匹配教師模型的logits，這造成了一個兩難困境：相似的教師模型幾乎無法提供新知識，而差異較大的教師模型則常給出無效引導，使得OPD大致局限於同族系內部的蒸餾。我們提出蒸餾強化學習（Distilled RL），該方法將教師監督整合進RL目標中，以提供細粒度的引導，選擇性地傳遞新知識並避免無條件模仿。Distilled RL包含三個組成部分：帶裁剪的逆向重要性抽樣、負樣本重置以及序列層級幾何正規化。透過一個簡潔且可解釋的案例研究，我們證明Distilled RL能有效將教師模型中先前無法獲取的知識傳遞給學生模型。在跨同族系與跨族系蒸餾場景的大量實驗中，Distilled RL在pass@1與pass@k指標上均顯著優於標準RL與OPD。我們的程式碼已於 https://github.com/597358816/Distilled-RL 公開。

English

Large language model (LLM) post-training is essential for improving reasoning, adaptation, and alignment. Existing methods mainly follow two paradigms: reinforcement learning (RL) and on-policy distillation (OPD). However, RL relies on coarse-grained outcome supervision, resulting in difficult credit assignment and limited capability to acquire new knowledge. OPD, meanwhile, unconditionally matches teacher logits through KL divergence, which creates a dilemma: similar teachers provide little new knowledge, while substantially different teachers often yield ineffective guidance, largely restricting OPD to within-family distillation. We propose Distilled Reinforcement Learning (Distilled RL), which integrates teacher supervision into the RL objective to provide fine-grained guidance, selectively transfer new knowledge and avoid unconditional imitation. Distilled RL contains three components: reverse importance sampling with clipping, negative sample reset, and sequence-level geometric normalization. Through a concise and interpretable case study, we demonstrate that Distilled RL can effectively transfer previously unavailable knowledge from a teacher model to a student model. Extensive experiments across both within-family and cross-family distillation settings show that Distilled RL substantially outperforms standard RL and OPD in terms of both pass@1 and pass@k. Our code is available at https://github.com/597358816/Distilled-RL.