超越模仿：基于推理进展的在线策略蒸馏过滤

摘要

在线策略蒸馏（OPD）已成为大语言模型后训练的一种有效框架，其通过将学生模型生成的轨迹与教师模型提供的密集token级监督相结合来实现。然而，OPD隐含地假设了教师派生奖励是推理进展的恰当代理，因此在策略优化过程中对所有教师反馈一视同仁。但在实践中，这一假设并非总是成立。我们观察到，教师派生奖励常常与真实推理进展相冲突，因为具有明确推理推进的推理步骤仍可能仅因其与教师输出的偏差而获得较低的蒸馏奖励。为解决这一错配问题，我们提出了面向在线策略蒸馏的推理进展感知奖励过滤方法（R2-OPD），该方法在轨迹内构建两组推理片段排序，一组来自教师派生奖励，另一组来自独立估计的进展奖励。当两组排序不一致时，蒸馏奖励会被选择性地抑制，从而减少与推理进展相冲突的监督，同时保留有效的教师指导。实验结果表明，我们的方法相较于标准OPD具有一致的性能提升，尤其在推理性能方面表现突出。

English

On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.