從自身錯誤中學習：構建可學習的微觀反思軌跡以進行自我蒸餾

摘要

自蒸馏通过使用模型自身的采样轨迹作为训练信号来提升大语言模型的推理能力，通常采用隐式logit层面对齐的方式，最小化与特权目标分布之间的KL散度。然而，由于这种监督信号通过无控采样生成，它既无法提供对模型具体错误的诊断性见解，也无法针对其个体失败模式给出修正性指导。因此，模型学习到的是模仿一种特权分布，而非接收细粒度修正——这类修正能够精准定位其推理失败的位置与原因。本文提出轨迹增强策略优化（TAPO），该方法将自蒸馏从隐式分布对齐提升为显式轨迹构建。在强化学习训练中，模型对同一查询同时产生正确与错误的采样轨迹，TAPO利用这种对比结构构建微反射修正：新的训练轨迹保留模型在失败点之前的错误推理，随后插入基于同一采样组中正确参考引导的自然语言诊断与修正推理。由于每条轨迹都锚定在学习者自身的前缀与解答上，相比基于KL散度的方法所施加的位置级对齐，这种修正信号能在更大程度上保留模型在策略分布的完整性。为整合这些轨迹，TAPO在模型能力边界处引入难度感知候选选择，并采用解耦优势估计以防止梯度污染。在AIME 2024、AIME 2025和HMMT 2025上的实验表明，在相同训练步数下，TAPO相较于GRPO取得了持续改进。进一步分析显示，TAPO同时增强了首轮推理能力与错误修正效果。

English

Self-distillation improves reasoning in large language models by using the model's own rollouts as training signal, typically through implicit logit-level alignment that minimizes KL divergence toward a privileged target distribution. However, because this supervision is generated via uncontrolled sampling, it provides no diagnostic insight into the model's specific errors or corrective guidance for its individual failure patterns. Consequently, the model learns to imitate a privileged distribution rather than receiving fine-grained corrections that pinpoint where and why its reasoning fails. In this paper, we propose Trajectory-Augmented Policy Optimization (TAPO), which advances self-distillation from implicit distributional alignment to explicit trajectory construction. During RL training, the model produces both correct and incorrect rollouts to the same query, and TAPO leverages this contrastive structure to construct micro-reflective corrections, new training trajectories that retain the model's erroneous reasoning up to the point of failure, then insert a natural-language diagnosis and corrected reasoning guided by a correct reference from the same sampling group. Since each trajectory is anchored in the learner's own prefix and solutions, the corrective signal preserves the model's on-policy distribution to a greater extent than the position-wise alignment imposed by KL-based methods. To integrate these trajectories, TAPO introduces difficulty-aware candidate selection at the model's capability boundary and decoupled advantage estimation to prevent gradient contamination. Experiments on AIME 2024, AIME 2025, and HMMT 2025 show that TAPO achieves consistent improvements over GRPO under the same number of training steps. Further analysis demonstrates that TAPO strengthens both first-pass reasoning and error-correction effectiveness.