TurnSight：面向工具集成推理的轮级事后自蒸馏

摘要

工具集成推理（TIR）使大语言模型（LLM）能够通过迭代式工具交互解决复杂任务。然而，现有的强化学习方法通常依赖轨迹级监督，限制了长视界TIR场景中的细粒度信用分配。同策略自蒸馏通过具有特权上下文的教师分支提供更密集的信号，但现有方法通常从真实答案或检索技能中获取此类上下文，这可能无法反映智能体实际访问的状态。此外，词元级监督无法捕捉工具交互的轮次级结构。为了解决这些问题，我们提出了TurnSight，一种轮次级事后自蒸馏框架，直接从执行条件化的事后信息中推导监督信号。随后，它构建多个具有不同前瞻视界的事后视角，并通过跨视界方向一致性选择可靠的监督。最后，选中的事后信号在兄弟轨迹之间进行归一化，并用于自适应地调节RL优势，同时保持其原始优化方向。在三个基准上的大量实验证明了TurnSight的有效性。我们的代码可在 https://github.com/quchangle1/TurnSight 获取。

English

Tool-Integrated Reasoning (TIR) enables LLMs to solve complex tasks through iterative tool interactions. However, existing reinforcement learning methods often rely on trajectory-level supervision, limiting fine-grained credit assignment in long-horizon TIR scenarios. On-policy self-distillation offers denser signals through teacher branches with privileged context, but existing approaches typically derive such context from ground-truth answers or retrieved skills, which may not reflect the states actually visited by the agent. Moreover, token-level supervision fails to capture the turn-level structure of tool interactions. To address this, we propose TurnSight, a turn-level hindsight self-distillation framework that derives supervision directly from execution-conditioned hindsight. It then constructs multiple hindsight views with different lookahead horizons and selects reliable supervision through cross-horizon directional agreement. Finally, the selected hindsight signal is normalized across sibling rollouts and used to adaptively modulate RL advantages while preserving their original optimization direction. Extensive experiments on three benchmarks demonstrate the effectiveness of TurnSight. Our codes are available at https://github.com/quchangle1/TurnSight.