SimpleOPD：面向长上下文推理的简单分词器无关同策略蒸馏

摘要

同策略蒸馏（OPD）为从更强的教师模型迁移推理能力提供了一条有前景的途径，但将其应用于长上下文推理教师模型与短上下文学生模型时，会引入实际挑战，包括分词器不匹配、教师-学生分布不匹配、响应长度爆炸以及训练不稳定。在本工作中，我们通过将证明推理能力从长上下文推理模型SU-01迁移至短上下文学生模型来研究该设置。为处理分词器差异，我们在共享文本空间中执行OPD，并且仅对齐在学生与教师分词器下占据完全相同文本跨度的token。为缓解生成长度过长和频繁截断的问题，我们引入了学生参考KL散度损失，并对特殊终止符（如</think>和<|im_end|>）的优势值进行掩蔽。该策略约束学生不过度偏离其初始策略，从而缓解教师-学生分布不匹配问题，并促进生成长度的稳定增长。在包括Qwen3、Qwen3.5、Intern-S2、GLM-4.7、Gemma-4在内的同族与异族学生模型上开展的实验表明，数学推理能力（尤其是自然语言数学证明）取得了一致提升。值得注意的是，Intern-S2-Preview在ProofBench上提升了21.2分，达到55.2，超越了Gemini-2.5-Pro。此外，HLE和HiPhO等科学基准上的表现也有所提升，表明OPD迁移的推理能力能够泛化到数学训练领域之外。

English

On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models, but applying it to long-context reasoning teachers and short-context students introduces practical challenges, including tokenizer mismatch, teacher-student distribution mismatch, response length explosion, and training instability. In this work, we study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models. To handle tokenizer differences, we perform OPD in a shared text space and align only tokens that occupy identical text spans under the student and teacher tokenizers. To mitigate the problem of excessive generation length and frequent truncation, we introduce a student reference KL loss and mask the advantages of special termination tokens such as </think> and <|im_end|>. This strategy constrains the student from drifting excessively from its initial policy, thereby mitigating the teacher-student distribution mismatch problem and fostering steady length growth. Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning, especially natural-language math proving. Notably, Intern-S2-Preview improves by 21.2 points on ProofBench, reaching 55.2 and surpassing Gemini-2.5-Pro. It also improves on science benchmarks such as HLE and HiPhO, suggesting that OPD transfers reasoning capabilities that generalize beyond the mathematical training domain.