AgentOPSD：用於智能體強化學習的遞迴自蒸餾

摘要

強化學習（RL）搭配可驗證獎勵會建構軌跡層級的優勢估計，但在長時程、多輪智慧體任務中，往往無法將成果歸因於少數決定成敗的關鍵決策。近期研究引入特權自蒸餾來進行信用分配，提供更密集的監督，然而此類局部訊號應如何表示序列信用仍不明確。我們提出 AgentOPSD，一種無評論家、遞迴式的回合層級信用分配方法，應用於智慧體強化學習。AgentOPSD 將詞元層級的師生對數機率差距聚合為回合層級證據，並在對數勝算空間中遞迴更新貝氏信念狀態。這產生了一套具有理論依據的重新加權機制，將稀疏的結果監督轉化為回合層級的信用訊號，並透過連續狀態之間的邊際信念修正來識別關鍵回合。該方法與標準策略最佳化完全相容，既不需要額外的評論家，也不需要額外的軌跡採樣。我們使用 Qwen2.5 模型在 ALFWorld、WebShop 和 Search-QA 上，以兩種規模（3B 和 7B）評估 AgentOPSD。AgentOPSD 優於 GRPO 和強自蒸餾基線，在使用 Qwen2.5-7B 時於 ALFWorld 上達到 89.1% 的成功率。消融研究將效能提升歸因於回合層級聚合與依賴歷史的遞迴信念更新。

English

Reinforcement learning (RL) with verifiable rewards constructs trajectory-level advantage estimates, yet it often fails to credit the few pivotal decisions that determine outcomes in long-horizon, multi-turn agentic tasks. Recent work introduces privileged self-distillation for credit assignment, providing denser supervision, but it remains unclear how such local signals should represent sequential credit. We propose AgentOPSD, a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning. AgentOPSD aggregates token-level teacher-student log-probability gaps into turn-level evidence and recursively updates a Bayesian belief state in log-odds space. This yields a principled reweighting scheme that converts sparse outcome supervision into turn-level credit signals and identifies pivotal turns through the marginal belief revision between consecutive states. The method is fully compatible with standard policy optimization and requires neither an additional critic nor extra rollouts. We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B). AgentOPSD outperforms GRPO and strong self-distillation baselines, achieving 89.1% success on ALFWorld with Qwen2.5-7B. Ablation studies attribute the gains to turn-level aggregation and history-dependent recursive belief updates.