接力傳遞：基於軌跡中繼的在策略蒸餾

摘要

在策略蒸馏（OPD）将词元级别的监督建立在学生自身轨迹之上，却受困于前缀失败问题：一旦学生走向错误推理方向，后续所有生成都建立在该偏差之上，产生方向错误的延续内容，不仅引发不可靠的监督，还浪费计算资源。我们识别出失败前缀上存在师生延续不对称性——教师倾向于重新引导方向，而学生则沿原方向继续——并将其转化为接力在策略蒸馏（Relay-OPD）中无标签的交接触发信号。训练时，Relay-OPD 通过让教师在检测到的触发点短暂接管以生成一段教师轨迹，随后学生继续生成并在整条接续轨迹上接受优化，从而构建接力轨迹。有限的接力预算将干预集中在关键的早期位置，同时限制与学生策略的偏离。以 Qwen3-4B-Instruct-2507 为教师、Qwen3-0.6B/1.7B-Non-Thinking 为学生模型，在八个数学推理基准上，Relay-OPD 在每个基准上均取得最佳或次佳结果，在 1.7B 模型上平均超越标准 OPD 5.73%、超越最强基线 FastOPD 1.49%，在 0.6B 模型上亦持续取得提升。训练轨迹长度减少超过 50%。

English

On-policy distillation (OPD) grounds token-level supervision in the student's own trajectory, yet suffers from prefix failure: once the student commits to a wrong reasoning direction, all subsequent generation builds on this deviation, producing misdirected continuations that elicit unreliable supervision and waste compute. We identify a teacher-student continuation asymmetry on failed prefixes, where the teacher tends to redirect while the student continues along the original direction, and convert it into a label-free handoff trigger in Relay On-Policy Distillation (Relay-OPD). During training, Relay-OPD constructs relay trajectories by letting the teacher briefly take over at detected trigger points to produce a teacher leg, after which the student resumes and is optimized on the resulting trajectory. A limited relay budget concentrates intervention on critical early positions while limiting departure from the student policy. With a Qwen3-4B-Instruct-2507 teacher and Qwen3-0.6B/1.7B-Non-Thinking students on eight mathematical reasoning benchmarks, Relay-OPD achieves the best or second-best results on every benchmark, outperforming standard OPD by +5.73% and the strongest baseline FastOPD by +1.49% on average for 1.7B, with consistent gains at 0.6B. Training trajectory length is reduced by over 50%.