超越模仿：基於推理進展的在策略蒸餾篩選

摘要

同策略蒸餾（OPD）已成為語言模型後訓練的有效框架，其做法是將學生生成的軌跡與教師提供的密集詞元級別監督配對。然而，OPD 隱含地假設教師衍生的獎勵是推理進展的適當代理指標，因此在策略優化過程中對所有教師回饋一視同仁。但在實務上，此一假設並非總是成立。我們觀察到，教師衍生的獎勵往往與真實的推理進展產生衝突，因為具有明確推理推進的推理步驟，仍可能僅因偏離教師的輸出而獲得較低的蒸餾獎勵。為了解決此一不一致性，我們提出適用於同策略蒸餾的推理進展感知獎勵過濾方法（R2-OPD），該方法在軌跡內建構兩組推理片段排序，一組來自教師衍生的獎勵，另一組則來自獨立估計的進展獎勵。每當兩組排序不一致時，蒸餾獎勵會被選擇性地抑制，從而減少與推理進展相衝突的監督，同時保留有效的教師引導。我們的實驗方法在標準 OPD 的基礎上展現出持續的改進，尤其在推理表現方面。

English

On-policy distillation (OPD) has emerged as an effective framework for post-training language models by pairing student-generated trajectories with dense token-level supervision from a teacher. However, OPD implicitly assumes that teacher-derived rewards are an appropriate proxy for reasoning progress, and therefore treats all teacher feedback equally during policy optimization. While in practice, this assumption does not always hold. We observe that teacher-derived rewards often conflict with genuine reasoning progress, as reasoning steps with clear reasoning advancement may still receive lower distillation rewards, simply due to deviation from teacher's outputs. To address this mismatch, we propose Reasoning-Progress-Aware Reward Filtering for On-Policy Distillation (R2-OPD), which constructs two within-trajectory rankings of reasoning spans, one from teacher-derived rewards and the other from independently estimated progress reward. Distillation rewards are selectively suppressed whenever the two rankings disagree, reducing supervision that conflicts with reasoning progress while preserving effective teacher guidance. Our approach shows consistent improvement over standard OPD especially regarding reasoning performances.