DRACO：基于动态评分标准的细粒度信用分配以支持长时程智能体训练

摘要

基于可验证奖励的强化学习在任务具有程序化检查器时效果良好，但大多数长程智能体领域并不具备此类检查器。我们研究的是结果不可观测(outcome-blind)的设置，其中无法获得真实成功信号。多标准评分标准(multi-criteria rubrics)是提供此类奖励的常用方式；它们每条轨迹仅评分一次，但单一标量在跨越数十步的决策过程中是较弱的信号。我们提出DRACO：面向优势分配的评分标准分布式优势(Distributing Rubric-based Advantage for Credit Optimization)。它在训练过程中动态生成评分标准以跟踪策略不断演进的能力，对每条完成的轨迹仅评分一次，并将该评判结果重新分配至与已标注评分标准相关的步骤上，从而在GRPO中产生差异化的逐步骤优势。重新分配过程为闭式求解，且不引入任何经过训练的属性归因模块。在AppWorld上，DRACO相较基础模型提升15.9个点，相较使用稀疏真实奖励训练的GRPO提升5.3个点，且自身未使用任何验证器。在分布外Tau-Bench上，即使没有前沿评判器，它也相较基础模型提升5.3个点，同时优于基于真实奖励的训练以及其他基于评分标准的训练设置。DRACO的代码可在 https://github.com/IBM/draco 获取。

English

Reinforcement Learning from Verifiable Rewards works well when a task has a programmatic checker, but most long-horizon agent domains have none. We work in the outcome-blind setting, where ground-truth success signals are not available. Multi-criteria rubrics are a popular way to supply such a reward; they are scored once per trajectory, but a single scalar is a poor signal across tens of steps. We propose DRACO: Distributing Rubric-based Advantage for Credit Optimization. It generates rubrics dynamically during training to track the policy's evolving capability, scores those rubrics once per completed trajectory, and redistributes that judgment over the steps responsible for annotated rubrics to produce differentiated per-step advantages in GRPO. The redistribution is closed-form and does not introduce any trained attribution module. On AppWorld, DRACO gains 15.9 points over the base model and 5.3 points over GRPO trained with a sparse ground-truth reward, despite not using any verifiers itself. On out-of-domain Tau-Bench, it gains 5.3 points over the base model even without a frontier judge, beating both ground-truth-reward training and other rubric-based training settings. The code for DRACO is available at https://github.com/IBM/draco.