AgentOPSD：面向智能体强化学习的递归自蒸馏方法

摘要

具有可验证奖励的强化学习（RL）会构建轨迹级优势估计，然而在长视界、多轮智能体任务中，它往往无法对决定结果的少数关键决策进行信用分配。近期工作引入了用于信用分配的特权自蒸馏，以提供更密集的监督，但这类局部信号应如何表示序列信用仍不明确。我们提出 AgentOPSD，一种用于智能体强化学习中回合级信用分配的无评论家递归方法。AgentOPSD 将 token 级教师-学生对数概率差距聚合为回合级证据，并在对数几率空间中递归更新贝叶斯信念状态。这产生了一种有理论依据的重新加权方案，可将稀疏的结果监督转化为回合级信用信号，并通过相邻状态之间的边际信念修正来识别关键回合。该方法与标准策略优化完全兼容，既不需要额外的评论家，也不需要额外的 rollout。我们使用 Qwen2.5 模型在两种规模（3B 和 7B）下，在 ALFWorld、WebShop 和 Search-QA 上评估了 AgentOPSD。AgentOPSD 优于 GRPO 和强自蒸馏基线，在 ALFWorld 上使用 Qwen2.5-7B 取得了 89.1% 的成功率。消融实验将性能提升归因于回合级聚合和历史依赖的递归信念更新。

English

Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks. Recent work introduces privileged self-distillation for credit assignment, providing denser supervision, but it remains unclear how such local signals should represent sequential credit. We propose AgentOPSD, a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state in log-odds space. This yields a principled reweighting scheme that converts sparse outcome supervision into turn-level credit signals and identifies pivotal turns through the marginal belief revision between consecutive states. The method is fully compatible with standard policy optimization and requires neither an additional critic nor extra rollouts. We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B). AgentOPSD outperforms GRPO and strong self-distillation baselines, achieving 89.1% success on ALFWorld with Qwen2.5-7B. Ablation studies attribute the gains to turn-level aggregation and history-dependent recursive belief updates.