PCSD：智能体强化学习中自蒸馏的持久一致性

摘要

大型语言模型智能体在复杂交互任务中展现出强大潜力，但其强化学习（RL）常因稀疏奖励而受阻，因为一条长的多轮轨迹可能仅获得单一的结果级信号。在策略自蒸馏（OPSD）通过特权教师提供稠密的词元级监督，但教师并非在每个位置都可靠。现有方法通常依赖孤立的词元级差异，这易受噪声影响，或分配统一的步级权重而忽视位置差异。我们提出持续一致性自蒸馏（PCSD），该方法从教师偏好信号的局部持续性中推导词元级蒸馏权重。PCSD将自适应窗口与指数衰减聚合相结合以捕获持续的相对教师支持，应用趋势感知调制来衰减局部下降的支持，并通过sigmoid门控产生连续权重。所得目标函数与GRPO联合优化，将稠密教师引导与稀疏环境反馈相结合。无需推理时技能，PCSD在两个骨干网络上均取得所有基线中最佳的ALFWorld Overall结果，超过GRPO 15.6和13.3个百分点，超过SDAR 6.2和5.5个百分点，同时在WebShop上保持竞争力，并在未见过的ALFWorld分割上较GRPO提升15.8个百分点。

English

Large language model agents have shown strong potential in complex interactive tasks, yet their reinforcement learning (RL) is often hindered by sparse rewards, as a long multi-turn trajectory may receive only a single outcome-level signal. On-policy self-distillation (OPSD) provides dense token-level supervision from a privileged teacher, but the teacher may not be reliable at every position. Existing methods commonly rely on isolated token-level discrepancies, which can be sensitive to noise, or assign a shared step-level weight that may overlook positional variation. We propose Persistent Consistency Self-Distillation (PCSD), which derives token-level distillation weights from the local persistence of teacher-favoring signals. PCSD combines adaptive windows with exponentially decayed aggregation to capture persistent relative teacher support, applies trend-aware modulation to attenuate locally declining support, and produces continuous weights through sigmoid gating. The resulting objective is jointly optimized with GRPO, combining dense teacher guidance with sparse environmental feedback. Without inference-time skills, PCSD achieves the best ALFWorld Overall results among all baselines on both backbones, exceeding GRPO by 15.6 and 13.3 points and SDAR by 6.2 and 5.5 points, while remaining competitive on WebShop and gaining 15.8 points over GRPO on unseen ALFWorld split.