TurnOPD：使在线策略蒸馏具备回合感知能力，实现长视界智能体高效训练

摘要

同策略蒸馏（OPD）通过让学生在自身轨迹上匹配更强的教师来训练学生策略，为语言智能体训练提供了有前景的框架。然而，其在长周期智能体任务中的应用尚未得到充分探索。我们识别出原始智能体OPD中的两个关键低效问题：（1）全周期展开通常将壁钟资源浪费在尾部轮次上，而尾部轮次提供的KL监督弱且噪声大；（2）轨迹级KL目标将大部分损失集中在浅层token上，一旦初始行为对齐后，深层决策轮次便得不到充分训练。为解决这些问题，我们提出TurnOPD，一种用于长周期智能体高效同策略蒸馏的回合级预算策略。TurnOPD包含两个预算控制器：自适应展开深度预算，利用基于探测的回合统计信息确定展开长度；以及渐进式回合归一化损失预算，将KL权重从token级逐步转向回合均衡监督。在ALFWorld、WebShop和Multi-Hop Search上使用任务特化教师模型进行的实验表明，在相同的壁钟训练预算下，TurnOPD实现了更优的验证准确率，并在准确率-时间前沿上超越了原始OPD。

English

On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD.