TRIAGE：角色類型化信用分配於自主代理強化學習

摘要

智能體強化學習需要對面嚮環境的動作（如搜索、點擊、編輯、導航指令和物件互動）進行信用分配。標準的GRPO將最終驗證器結果作為所有動作令牌的統一優勢值。此結果訊號有其用處，但在結構上不完整：它在失敗的軌跡中懲罰了有用的探索，並在成功的軌跡中強化了冗余或倒退的動作。我們提出TRIAGE，一種角色類型化的信用分配框架，為結果信用添加了語義角色軸。一個結構化裁判將每個片段分類為決定性進展、有用探索、無進展基礎設施或倒退，並通過固定的角色條件規則將這些標籤映射到有界的片段層級過程獎勵。這保持了驗證器結果作為優化方向的來源，同時修正了僅基於結果信用的兩個主要盲點。我們進一步證明，角色條件化的信用是僅從角色標籤可表達的最優片段層級修正——將每片段優勢殘差投影到角色變量上——從而只要裁判可靠，固定的角色常數就能降低優勢估計誤差，並將此與更低方差的策略梯度聯繫起來。在ALFWorld、Search-QA和WebShop上，TRIAGE在兩種策略模型上的成功率高於GRPO，並且優於標量裁判推導的過程獎勵和以結果監督的共享主幹價值基線。消融實驗表明，增益來自角色類型化，而非僅僅添加密集獎勵：在成功軌跡內可靠檢測倒退是主要貢獻因素，而探索信用則提供一致的次要增益；在已完成的ALFWorld和WebShop軌跡中，相較於GRPO，TRIAGE還分別減少了10.4%和14.8%的面嚮環境回合數。

English

Agentic reinforcement learning requires assigning credit to environment-facing actions such as searches, clicks, edits, navigation commands, and object interactions. Standard GRPO uses the final verifier outcome as a uniform advantage over all action tokens. This outcome signal is useful but structurally incomplete: it punishes useful exploration in failed rollouts and reinforces redundant or regressive actions in successful rollouts. We propose TRIAGE, a role-typed credit assignment framework that adds a semantic role axis to outcome credit. A structured judge classifies each segment as decisive progress, useful exploration, no-progress infrastructure, or regression, and a fixed role-conditioned rule maps these labels to bounded segment-level process rewards. This keeps verifier outcomes as the source of optimization direction while correcting the two main blind spots of outcome-only credit. We further show that role-conditioned credit is the optimal segment-level correction expressible from role labels alone -- a projection of the per-segment advantage residual onto the role variable -- so that the fixed role constants reduce advantage estimation error whenever the judge is reliable, and we connect this to lower-variance policy gradients. Across ALFWorld, Search-QA, and WebShop, TRIAGE improves success rates over GRPO for two policy models and outperforms both a scalar judge-derived process reward and an outcome-supervised shared-backbone value baseline. Ablations show that the gain comes from role typing rather than merely adding dense rewards: reliable detection of regression inside successful trajectories is the dominant contributor, while exploration credit provides a consistent secondary gain; on completed ALFWorld and WebShop rollouts, TRIAGE also reduces environment-facing turns by an additional 10.4% and 14.8% relative to GRPO.