CoRT：令牌级评分引导策略优化的反事实重放

摘要

基于评分标准的强化学习通过依据显式标准评估模型输出来丰富语言模型训练。然而，在GRPO（分组相对策略优化）风格的流程中，这些结构化判断被简化为标量的响应级奖励，并转化为响应级优势，统一广播到所有生成的token上。这导致即使在不同的标准基于不同的区间、格式决策或语义选择时，也没有显式的机制在响应内部进行信用分配。我们提出CoRT，一种用于评分标准条件GRPO的token级信用加权方法。CoRT不训练辅助的token评分模型，而是通过反事实重放，在原始评分标准条件的提示和匹配的无标准提示下，对同一采样响应重新评分。由此产生的token级对数似然对比作为对评分标准上下文依赖程度的代理指标。CoRT将这些对比映射到有界、响应归一化的权重上，并利用这些权重在token间重新分配带符号的GRPO优势，无需引入辅助评分器或改变响应级奖励。在指令调优模型和不同奖励粒度上的实验表明，CoRT在绝大多数比较中优于匹配的响应级GRPO，平均提升4.4个百分点。该方法与学习型token级信用基线相比仍具有竞争力，同时避免了独立的关联性学习阶段。这些结果表明，策略内部的反事实似然对比能够提供有效的训练信号用于响应内信用分配，同时保持GRPO的简洁性和稳定性。

English

Rubric-based reinforcement learning enriches language model training by evaluating model outputs against explicit criteria. Yet in GRPO-style pipelines, these structured judgments are reduced to a scalar response-level reward and converted into a response-level advantage, which is broadcast uniformly to all generated tokens. This leaves no explicit mechanism for allocating credit within a response, even when different criteria are grounded in different spans, formatting decisions, or semantic choices. We propose CoRT, a token-level credit weighting method for rubric-conditioned GRPO. Instead of training an auxiliary token scoring model, CoRT uses counterfactual replay to rescore the same sampled response under the original rubric-conditioned prompt and a matched criteria-free prompt. The resulting tokenwise log-likelihood contrasts serve as a proxy for dependence on the rubric context. CoRT maps these contrasts to bounded, response-normalized weights and uses them to redistribute the signed GRPO advantage across tokens, without introducing an auxiliary scorer or changing the response-level reward. Experiments across instruction-tuned models and reward granularities show that CoRT improves over matched response-level GRPO in the vast majority of comparisons, with an average gain of 4.4 percentage points. The method remains competitive with learned token-level credit baselines while avoiding a separate relevance-learning stage. These results suggest that policy-internal counterfactual likelihood contrasts provide an effective training signal for within-response credit allocation while retaining the simplicity and stability of GRPO.