PBSD：面向长周期信用分配的特权贝叶斯自蒸馏

摘要

长时域智能体任务对基于结果的强化学习提出了一个根本性的信用分配挑战：轨迹级奖励只能验证最终结果的正确性，却难以指明哪些中间推理步骤或工具交互对结果产生了贡献。这一困难在多轮搜索智能体中尤为突出——成功轨迹可能包含误导性动作，而失败轨迹也可能包含有价值的证据收集步骤。我们提出PBSD（特权贝叶斯自蒸馏），一种在稀疏最终奖励下进行细粒度信用分配的贝叶斯校准自蒸馏方法。PBSD通过已验证答案的后验-先验概率比来衡量轨迹质量，并利用贝叶斯法则将这一难以估计的答案侧比率转换为标准学生模型与特权答案条件教师模型之间易于处理的似然比。通过对该贝叶斯证据分数进行自回归分解，得到能够识别每个中间轮次是支持还是削弱已验证结果的轮次级信号。因此，PBSD提供了一种原则性且优雅的重新加权方案，将稀疏的结果监督转化为经过贝叶斯校准的轮次级信用信号，同时与标准策略优化保持完全兼容。实验表明，PBSD在领域内和领域外场景中均能持续提升性能，并能有效将知识从短上下文训练迁移到长上下文推理中，表明其细粒度信用分配机制有助于更有效的策略学习，并带来更好的泛化能力。

English

Long-horizon agentic tasks pose a fundamental credit assignment challenge for outcome-base reinforcement learning: trajectory-level rewards verify final correctness but provide limited guidance on which intermediate reasoning steps or tool interactions contribute to the outcome. The difficulty is especially pronounced in multi-turn search agents, where successful trajectories may contain misleading actions and failed trajectories may contain valuable evidence-gathering steps. We propose PBSD (Privileged Bayesian Self-Distillation), a Bayes-calibrated self-distillation method for fine-grained credit assignment under sparse final rewards. PBSD measures trajectory quality through the posterior-to-prior probability ratio of the verified answer and applies Bayes' rule to convert this hard-to-estimate answer-side ratio into a tractable likelihood ratio between a standard student model and a privileged answer-conditioned teacher model. Autoregressive decomposition of this Bayesian evidence score yields turn-level signals that identify whether each intermediate turn supports or undermines the verified outcome. Consequently, PBSD provides a principled and elegant reweighting scheme that transforms sparse outcome supervision into Bayes-calibrated turn-level credit signals, while remaining fully compatible with standard policy optimization. Experiments demonstrate that PBSD consistently enhances performance across both in-domain and out-of-domain settings, and effectively transfers knowledge from short-context training to long-context inference, suggesting that its fine-grained credit assignment mechanism facilitates more effective policy learning and yields improved generalization.