ReNIO：針對大型語言模型在線策略蒸餾的負面軌跡重要性重新加權

摘要

同策略蒸馏（OPD）通过让学生模型基于自身生成输出进行训练以提升大语言模型的推理能力，但标准OPD平等对待所有学生生成输出（SGOs），未考虑其信息量差异。我们在受控过滤实验中观察到一致的不对称性：在OPD与同策略自蒸馏（OPSD）中，仅使用错误SGOs进行训练的效果优于仅使用正确SGOs。进一步分析表明，仅用正确SGOs训练的模型倾向于生成更短的推理轨迹，且反思行为较弱；而错误SGOs则能更好地保留接近模型能力边界的探索性推理。为利用这一信号而不需要包含完整答案的轨迹生成，我们提出ReNIO（Reweights Negative trajectory Importance for LLM On-policy distillation），即针对大语言模型同策略蒸馏的负轨迹重要性重加权。ReNIO通过学生与教师概率之比，识别导致错误推理轨迹的关键词元，并将其信息聚合为归一化的样本权重，从而在无需观察最终答案正确性的情况下，内在性地将更大权重分配给可能的负轨迹。由于Re-NIO仅使用前缀条件下的词元概率，它保留了OPD相对于完整轨迹强化学习的前缀训练优势。在数学推理与代码生成任务中，ReNIO同时改进了OPD与OPSD，在数学推理基准上，Qwen3-1.7B与R1-Distill-Qwen-7B分别实现了高达8.90%与10.00%的代表性相对增益。代码仓库：https://github.com/BDML-lab/ReNIO。

English

On-policy distillation (OPD) improves LLM reasoning by training a student model on its own generated outputs, but standard OPD treats all student-generated outputs (SGOs) equally regardless of their informativeness. We observe a consistent asymmetry in controlled filtering experiments: in both OPD and on-policy self distillation (OPSD), training only on incorrect SGOs outperforms training only on correct ones. Our further analysis suggests that models trained on correct-only SGOs tend to generate shorter reasoning traces and show weaker reflection behavior, while incorrect SGOs better preserve exploratory reasoning near the model's capability boundary. To exploit this signal without requiring full answer-containing rollouts, we introduce ReNIO, which Reweights Negative trajectory Importance for LLM On-policy distillation. By using the student-to-teacher probability ratio, ReNIO identifies pivotal tokens leading to wrong reasoning traces and aggregates their information into a normalized sample weight, inherently assigning larger weights to likely negative trajectories without observing the correctness of final-answer. Since Re-NIO only uses prefix-conditioned token probabilities, it preserves OPD's prefix training advantage over full-rollout reinforcement learning. Across both mathematical reasoning and code generation tasks, ReNIO improves both OPD and OPSD, with representative relative gains of up to 8.90% for Qwen3-1.7B and 10.00% for R1-Distill-Qwen-7B on mathematical reasoning benchmarks. Code repo: https://github.com/BDML-lab/ReNIO.