TRIAGE：面向智能体强化学习的角色类型化信用分配

摘要

代理式强化学习需要对外向性动作（如搜索、点击、编辑、导航指令和对象交互）进行信用分配。标准GRPO方法将所有动作令牌的最终验证器结果视为统一优势值，这种结果信号虽有用但结构不完整：它会在失败轨迹中惩罚有效探索，并在成功轨迹中强化冗余或回溯性动作。我们提出TRIAGE框架，这是一种角色类型化的信用分配方法，通过添加语义角色维度扩展结果信用。结构化评判器将每个片段分类为决定性进展、有效探索、无进展基础设施或回溯性行为，并通过固定角色条件规则将这些标签映射为有界片段级过程奖励。该方法在保持验证器结果作为优化方向来源的同时，修正了纯结果信用分配的两个主要盲点。我们进一步证明，角色条件信用是从角色标签本身可推导的最优片段级修正——即每个片段优势残差在角色变量上的投影——因此当评判器可靠时，固定角色常数可降低优势估计误差，并与此相关的低方差策略梯度建立联系。在ALFWorld、Search-QA和WebShop测试中，TRIAGE相比GRPO提升了两种策略模型的成功率，且优于标量评判器推导的过程奖励和结果监督共享骨干价值基线。消融实验表明，性能提升源于角色类型化而非单纯增加密集奖励：成功轨迹内部回溯性行为的可靠检测是主要贡献因素，而探索信用提供稳定的次要增益；在ALFWorld和WebShop的完整轨迹中，TRIAGE相较GRPO额外减少10.4%和14.8%的外向性动作轮次。

English

Agentic reinforcement learning requires assigning credit to environment-facing actions such as searches, clicks, edits, navigation commands, and object interactions. Standard GRPO uses the final verifier outcome as a uniform advantage over all action tokens. This outcome signal is useful but structurally incomplete: it punishes useful exploration in failed rollouts and reinforces redundant or regressive actions in successful rollouts. We propose TRIAGE, a role-typed credit assignment framework that adds a semantic role axis to outcome credit. A structured judge classifies each segment as decisive progress, useful exploration, no-progress infrastructure, or regression, and a fixed role-conditioned rule maps these labels to bounded segment-level process rewards. This keeps verifier outcomes as the source of optimization direction while correcting the two main blind spots of outcome-only credit. We further show that role-conditioned credit is the optimal segment-level correction expressible from role labels alone -- a projection of the per-segment advantage residual onto the role variable -- so that the fixed role constants reduce advantage estimation error whenever the judge is reliable, and we connect this to lower-variance policy gradients. Across ALFWorld, Search-QA, and WebShop, TRIAGE improves success rates over GRPO for two policy models and outperforms both a scalar judge-derived process reward and an outcome-supervised shared-backbone value baseline. Ablations show that the gain comes from role typing rather than merely adding dense rewards: reliable detection of regression inside successful trajectories is the dominant contributor, while exploration credit provides a consistent secondary gain; on completed ALFWorld and WebShop rollouts, TRIAGE also reduces environment-facing turns by an additional 10.4% and 14.8% relative to GRPO.