H^2SD：混合回顧自我蒸餾

摘要

基于可验证奖励的强化学习（RLVR）在提升大语言模型推理能力方面取得了显著进展，特别是在数学推理和代码生成等任务上。然而，大多数RLVR方法为完整轨迹分配标量结果奖励，导致监督信号稀疏且词元级信用分配受限。在线策略蒸馏（OPD）通过从更强的教师模型中蒸馏词元级分布来提供更密集的监督，但需要额外的教师模型且通常假设共享词汇表。在线策略自蒸馏（OPSD）通过使同一模型基于特权信息构建教师策略来消除这一依赖。然而，直接匹配教师分布可能导致信息泄露和优化不稳定。RLSD通过仅利用教师信号调节更新幅度来避免直接匹配，但当采样推理失败时无法提供明确的修正方向。为应对这一权衡，我们提出H²SD——一种混合事后自蒸馏框架，根据轨迹正确性差异化使用教师信号。对于成功轨迹，教师接收经确认为正确的学生回答及重述指令，并利用其对原始回答词元的概率来调节更新幅度，而不改变由奖励确定的更新方向。对于失败轨迹，我们使教师基于包含关键推理步骤与已验证答案的参考提示进行条件生成，并最小化学生分布相对于教师分布的反向KL散度。在多个具有挑战性的推理基准上的实验表明，H²SD在保持稳定优化和高效生成的同时，持续优于代表性的RLVR、OPSD和RLSD基线方法。

English

Reinforcement learning with verifiable rewards (RLVR) has substantially improved the reasoning capabilities of large language models on tasks such as mathematical reasoning and code generation. However, most RLVR methods assign a scalar outcome reward to an entire trajectory, resulting in sparse supervision and limited token-level credit assignment. On-policy distillation (OPD) provides denser supervision by distilling token-level distributions from a stronger teacher model, but requires an additional teacher and typically assumes a shared vocabulary. On-policy self-distillation (OPSD) removes this dependency by conditioning the same model on privileged information to construct a teacher policy. However, directly matching the teacher distribution may cause information leakage and unstable optimization. RLSD avoids direct matching by using the teacher signal only to modulate update magnitudes, but it cannot provide an explicit correction direction when the sampled reasoning fails. To address this tradeoff, we introduce H^{2}SD, a hybrid hindsight self distillation framework that uses the teacher differently according to trajectory correctness. For successful trajectories, the teacher receives the student response confirmed as correct together with a rephrasing instruction, and its probabilities on the original response tokens are used to modulate update magnitudes without changing the direction determined by the reward. For failed trajectories, we condition the teacher on a reference hint containing key reasoning steps and a verified answer, and minimize the reverse KL divergence from the student to the teacher. Experiments on multiple challenging reasoning benchmarks show that H^2SD consistently outperforms representative RLVR, OPSD, and RLSD baselines while maintaining stable optimization and favorable generation efficiency.