SimpleOPD：適用於長上下文推理的簡易分詞器無關同策略蒸餾方法

摘要

同策略蒸餾（OPD）為將推理能力從更強的教師模型遷移而來提供了一種有前景的途徑，但將其應用於長上下文推理教師與短上下文學生的組合時，會引入實際挑戰，包括 tokenizer 不匹配、師生分布不匹配、回應長度爆炸以及訓練不穩定性。在本研究中，我們透過將證明推理能力從長上下文推理模型 SU-01 遷移至短上下文學生模型來探討此設定。為處理 tokenizer 差異，我們在共享文本空間中執行 OPD，並且僅對齊在學生與教師 tokenizer 下佔據相同文本跨度的 token。為緩解生成長度過長與頻繁截斷的問題，我們引入了學生參考 KL 損失，並遮罩諸如 `</think>` 和 `<|im_end|>` 等特殊終止 token 的優勢。此策略限制學生模型過度偏離其初始策略，從而緩解師生分布不匹配問題，並促使長度穩定增長。在包括 Qwen3、Qwen3.5、Intern-S2、GLM-4.7、Gemma-4 在內的同一系列與不同系列學生模型上的實驗，一致顯示出數學推理能力的提升，尤其是自然語言數學證明方面。值得注意的是，Intern-S2-Preview 在 ProofBench 上提升了 21.2 分，達到 55.2，並超越了 Gemini-2.5-Pro。它在 HLE 和 HiPhO 等科學基準上也有提升，表明 OPD 所遷移的推理能力具有超越數學訓練領域的泛化性。

English

On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models, but applying it to long-context reasoning teachers and short-context students introduces practical challenges, including tokenizer mismatch, teacher-student distribution mismatch, response length explosion, and training instability. In this work, we study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models. To handle tokenizer differences, we perform OPD in a shared text space and align only tokens that occupy identical text spans under the student and teacher tokenizers. To mitigate the problem of excessive generation length and frequent truncation, we introduce a student reference KL loss and mask the advantages of special termination tokens such as </think> and <|im_end|>. This strategy constrains the student from drifting excessively from its initial policy, thereby mitigating the teacher-student distribution mismatch problem and fostering steady length growth. Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning, especially natural-language math proving. Notably, Intern-S2-Preview improves by 21.2 points on ProofBench, reaching 55.2 and surpassing Gemini-2.5-Pro. It also improves on science benchmarks such as HLE and HiPhO, suggesting that OPD transfers reasoning capabilities that generalize beyond the mathematical training domain.