RISE：透過自我外推策略蒸餾實現的遞迴改進

摘要

同策略蒸餾（OPD）為語言模型後訓練提供密集的逐詞元監督，但其有效性受到教師品質的瓶頸限制：外部教師存在分佈不匹配的問題，而具有特權條件的自蒸餾則受限於上下文學習能力。我們提出 RISE（Recursive Improvement via Self-Extrapolating Policy Distillation，即透過自我外推策略蒸餾實現的遞迴改進），直接從模型自身的 RLVR 訓練軌跡中建構合成教師。透過外推當前檢查點與後方錨點之間在參數空間或輸出 logit 空間中的位移，RISE 將稀疏的、由結果誘發的參數更新轉化為密集的詞元級目標，無需任何外部模型或特權條件。RISE 以互補迴圈結合 RLVR 與 OPD：結果獎勵將外推錨定於正確的推理方向，而外推教師則細化詞元級決策。此外，由於教師會隨學生每次迭代的進步而不斷更新，蒸餾因而成為一種遞迴改進機制，而非一次性壓縮步驟。涵蓋數學推理、多領域 STEM、程式碼生成及多輪代理任務的實驗顯示，RISE 在所有設定下均優於僅使用 RLVR 的訓練與同策略自蒸餾。

English

On-policy distillation (OPD) provides dense, per-token supervision for language model post-training, but its effectiveness is bottlenecked by teacher quality: external teachers suffer from distribution mismatch, while self-distillation with privileged conditioning is limited by in-context learning capacity. We propose RISE (Recursive Improvement via Self-Extrapolating Policy Distillation), which constructs a synthetic teacher directly from the model's own RLVR training trajectory. By extrapolating the displacement between the current checkpoint and a trailing anchor---in parameter space or output logit space---RISE converts a sparse outcome-induced parameter update into a dense token-level target, without any external model or privileged conditioning. RISE combines RLVR and OPD in a complementary loop: outcome rewards ground the extrapolation toward correct reasoning, while the extrapolated teacher refines token-level decisions. Moreover, since the teacher is refreshed every iteration as the student improves, distillation becomes a recursive improvement mechanism rather than a one-shot compression step. Experiments spanning mathematical reasoning, multi-domain STEM, code generation, and multi-turn agentic tasks show that RISE outperforms RLVR-only training and on-policy self-distillation across all settings.