重新審視用於後訓練的完整推理軌跡

摘要

大型語言模型（LLMs）常透過在預先蒐集的推理軌跡上進行後訓練，以提升其推理能力。此類軌跡往往因路徑複雜交織而顯得冗長，且通常包含通往答案途中的繞路。然而，在後訓練（例如監督式微調（SFT））中，LLMs 是否確實能從學習完整軌跡中獲益，仍未受到充分探討。從我們的先導研究出發，我們發現完整軌跡僅帶來有限效益，而部分軌跡即使在大幅截斷下仍具效果。我們透過基於注意力的分析與受控的 token 移除研究來分析推理軌跡中的冗餘，兩者皆顯示中間 token 對最終推理品質的貢獻極小。這暗示，避免冗餘資訊可能讓 LLMs 在已知軌跡端點的情況下，藉由從內部知識推斷缺失步驟，在內部推論出連貫的替代方案。此外，我們顯示使用端點訓練 LLMs 會導致推理行為的一致變化，且此舉亦有益於基於強化學習或同策略（on-policy）蒸餾的後訓練方法，凸顯重新審視完整推理軌跡的必要性。程式碼可於 https://github.com/naver-ai/revisiting-trace 取得。

English

Large language models (LLMs) are often post-trained on pre-collected reasoning trajectories to improve their reasoning capability. Such trajectories tend to be long due to complex, interwoven paths, which often include detours on the path toward the answer. However, it has been underexplored whether LLMs indeed benefit from learning complete trajectories in post-training, such as supervised fine-tuning (SFT). Starting from our pilot study, we find that full trajectories provide only limited benefit, while partial trajectories are effective even under heavy truncation. We analyze redundancy in reasoning trajectories through attention-based analyses and controlled token-removal studies, both of which show that intermediate tokens contribute minimally to final reasoning quality. This suggests that avoiding redundant information may allow LLMs to internally infer coherent alternatives by inferring missing steps from their internal knowledge, given known trajectory endpoints. Furthermore, we show that training LLMs using endpoints leads to consistent changes in reasoning behavior, and that it also benefits post-training methods based on reinforcement learning or on-policy distillation, highlighting the need to revisit complete reasoning traces. Code is available at https://github.com/naver-ai/revisiting-trace.