在策略自蒸馏：无需任何监督

摘要

在策略（自我）蒸餾（OPD/OPSD）在大型語言模型（LLMs）的後期訓練中展現了強大的潛力。然而，現有方法仍高度依賴外部監督，包括真實標籤信號、環境回饋或更大模型的引導，因此無法實現真正的「自我」蒸餾。本研究中，我們證明僅透過模型自身的生成結果以及內部一致性，即可實現在策略自我蒸餾。我們提出無監督在策略自我蒸餾（U-OPSD）。U-OPSD 首先採樣多個 rollout，並在自我一致性閾值下透過多數決建構偽解決方案；接著，讓模型的分佈以偽解決方案為條件，並在不一致的生成結果上進行自我蒸餾，使模型能在其自信但錯誤之處進行精確修正。在各種基準測試、基礎模型與訓練設定下，U-OPSD 一致地優於基礎模型，並與使用真實標籤（GT）的監督方法（如 OPSD 和 GRPO）相當或更佳。在五個數學推理基準（AIME24、AIME25、HMMT25、MATH500 和 AMC23）上，U-OPSD 在 Qwen3 非思考模式下，於 4B 和 8B 規模分別比基礎模型提升 8.5% 和 10.7%，且平均分別領先 OPSD 3.2% 和 2.3%。在思考模式下，U-OPSD 與 OPSD 持平，4B 規模領先 0.9%，8B 規模持平，並分別超越 GRPO 0.7% 和 1.1%。程式碼位於 [https://github.com/williamium3000/u-opsd](https://github.com/williamium3000/u-opsd)。

English

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose unsupervised on-policy self-distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo solution by majority vote under a self-consistency threshold. It then conditions the model's distribution on the pseudo-solution and distills itself on the disagreeing completions, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT) such as OPSD and GRPO. On five mathematical reasoning benchmarks, i.e., AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales, and outperforms OPSD by 3.2% and 2.3% on average, respectively. In thinking mode, U-OPSD stays on par with OPSD, ahead by 0.9% at 4B and level at 8B and surpassing GRPO by 0.7% and 1.1%, respectively. Code is available at [https://github.com/williamium3000/u-opsd](https://github.com/williamium3000/u-opsd).