TurnOPD：使在线策略蒸馏具备回合感知能力，实现高效的长时域智能体训练

摘要

同策略蒸餾（OPD）透過在學生自身的軌跡上匹配更強教師的策略來訓練學生策略，為語言智能體的訓練提供了一個有前景的框架。然而，其在長程智能體任務中的應用仍有待充分探索。我們發現基礎智能體OPD存在兩個關鍵效率問題：（1）完整軌跡的滾動通常會浪費牆鐘資源在尾部回合上，而這些回合提供的KL監督既微弱又帶有雜訊；（2）軌跡層級的KL目標將大部分損失集中在淺層詞元上，一旦初始行為對齊後，更深層的決策回合便訓練不足。為解決這些挑戰，我們提出TurnOPD——一種針對長程智能體高效同策略蒸餾的回合級預算策略。TurnOPD包含兩個預算控制器：自適應滾動深度預算，利用基於探測的回合統計數據決定滾動長度；以及漸進式回合歸一化損失預算，逐步將KL加權從詞元級別轉向回合平衡的監督。在ALFWorld、WebShop以及使用任務專門化教師模型的多跳搜索任務上的實驗表明，在相等的牆鐘訓練預算下，TurnOPD實現了更優的驗證準確率，並將準確率-時間前沿推向超越基礎OPD。

English

On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD.