传递接力棒：轨迹中继的在线策略蒸馏

摘要

在线策略蒸馏（OPD）通过学生自身轨迹提供令牌级监督，但存在前缀失败问题：一旦学生陷入错误的推理方向，后续生成均建立在此偏差之上，产生方向错误的连续输出，导致监督信号不可靠且计算资源浪费。我们识别出在失败前缀上存在师生延续不对称现象——教师倾向于重定向推理路径，而学生则延续原始方向——并将其转化为中继在线策略蒸馏（Relay-OPD）中无需标签的切换触发机制。在训练过程中，Relay-OPD通过让教师在检测到的触发点短暂接管生成"教师段"，随后学生继续生成并基于完整轨迹进行优化，从而构建中继轨迹。有限的中继预算将干预集中在关键早期位置，同时限制对原学生策略的偏离。以Qwen3-4B-Instruct-2507为教师、Qwen3-0.6B/1.7B-Non-Thinking为学生模型，在八个数学推理基准上，Relay-OPD在每个基准均取得最优或次优结果：1.7B模型平均超越标准OPD达5.73%，超越最强基线FastOPD达1.49%；0.6B模型也保持稳定提升。训练轨迹长度减少超过50%。

English

On-policy distillation (OPD) grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute. We identify a teacher-student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD). During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates intervention on critical early positions while limiting departure from the student policy. With a Qwen3-4B-Instruct-2507 teacher and Qwen3-0.6B/1.7B-Non-Thinking students on eight mathematical reasoning benchmarks, Relay-OPD achieves the best or second-best results on every benchmark, outperforming standard OPD by +5.73% and the strongest baseline FastOPD by +1.49% on average for 1.7B, with consistent gains at 0.6B. Training trajectory length is reduced by over 50%.