多轮在线策略蒸馏与前缀回放

摘要

我们研究面向智能体任务的在策略蒸馏（OPD），其中大语言模型（LLM）智能体需与环境进行多轮交互，学生模型需模仿教师模型在这些多轮交互历史中的行为。完全在线的OPD成本高昂，因为每次更新都需要学生模型通过环境生成全新轨迹，并在访问过的历史状态上查询教师模型。为此，我们提出重放前缀在策略蒸馏（ReOPD），这是一种离环境替代方案，通过复用预收集的教师轨迹作为重放前缀：学生模型仅在选定步骤执行动作，而教师模型在不触发新环境交互的前提下提供密集的逐步骤监督。我们发现多轮OPD存在一个前缀陷阱：使历史状态更贴近学生模型的实际分布虽能提升与学生的相关性，但可能导致在教师模型目标不可靠的历史状态上发起查询，从而在学生占据分布与教师可靠性之间形成双向分布偏移。ReOPD通过将多轮OPD转化为可靠性感知的前缀分布设计来解决该问题，并采用简单的步长衰减采样调度方案实现——优先选取早期、偏移幅度较小的前缀。在数学推理与搜索环境中，跨越多个教师与学生模型规模的实验表明：ReOPD能保持或提升OPD级别的准确率，学生训练过程中零工具调用，且每次轨迹采样的速度至少是OPD的4倍。因此，ReOPD将昂贵的智能体-环境交互转化为可复用的离线资源，实现了跨工具、任务和环境的高效蒸馏。

English

We study on-policy distillation (OPD) for agentic tasks, where an LLM agent interacts with an environment over multiple turns and a student imitates a teacher over these multi-turn interaction histories. Fully online OPD is costly because each update requires fresh student rollouts through the environment and teacher queries at visited histories. We propose Replayed-Prefix On-Policy Distillation (ReOPD), an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes: the student acts at selected steps, while the teacher provides dense per-step supervision without executing new environment interactions. We show that multi-turn OPD introduces a prefix trap: making histories more student-on-policy improves relevance to the student, but can query the teacher on histories where its target is unreliable. This creates a two-sided distribution shift between student occupancy and teacher reliability. ReOPD addresses this by treating multi-turn OPD as a reliability-aware prefix distribution design and implements it with a simple step-decaying sampling schedule that emphasizes early, lower-shift prefixes. Across mathematical reasoning with Python and search environments over multiple teacher and student model scales, ReOPD preserves or improves OPD-level accuracy, uses zero tool calls during student training, and is at least 4times faster per rollout than OPD. ReOPD therefore turns expensive agent-environment interaction into a reusable offline resource, enabling scalable distillation across tools, tasks, and environments.