并非所有Token都值得同等信用：面向长思维链推理的反事实敏感性信用重分配

摘要

可验证奖励强化学习（RLVR）是提升大语言模型长思维链推理能力的核心方法。无评论家方法（如GRPO）将响应级奖励转化为优势值，并均匀地广播到所有词元上，忽视了各词元对最终结果的不平等贡献。相比之下，在线自蒸馏（OPSD）通过最小化无特权策略与特权自教师之间的前向KL散度来提供密集的分布级监督，其隐含假设是所产生的似然偏移编码了可靠的、与答案一致的信息。我们通过固定每条采样轨迹并在两种对立的输出条件下重新评分来检验这一前提：一种条件判定轨迹正确，另一种判定其错误。结果表明，大多数受影响词元在两种条件下的偏移方向相同，符号反转极少，且所引发的优化信号存在大量重叠。此外，大幅偏移集中在高度可替换的表层形式词元上，而承载问题特定推理内容的词元敏感度较低。这些发现表明，特权偏移无法提供可靠的、与答案一致的方向，其幅度主要反映反事实敏感度而非词元级学习价值。基于上述观察，我们提出反事实敏感度信用再分配（CSCR），这是GRPO的一种简单扩展，通过降低高度敏感词元的信用并重新归一化词元级优势值，以同时保留原始信用预算和验证器决定的方向。在长思维链数学推理基准上，CSCR在相同策略更新次数下持续优于GRPO基线。针对性消融实验进一步佐证了我们的诊断：特权引发的方向不可靠，适度降权最为有效，而更强的调制会破坏优化稳定性。

English

Reinforcement learning with verifiable rewards (RLVR) is central to improving long-CoT reasoning in large language models. Critic-free methods such as GRPO convert response-level rewards into advantages and uniformly broadcast them across tokens, overlooking their unequal contributions to the final outcome. On-policy self-distillation (OPSD) instead provides dense distributional supervision by minimizing the forward KL divergence between an unprivileged policy and a privileged self-teacher, implicitly assuming that the resulting likelihood shifts encode reliable answer-aligned information. We test this premise by fixing each sampled trajectory and re-scoring it under two opposing outcome conditions, one asserting correctness and the other incorrectness. Most affected tokens shift in the same direction under both conditions, with few sign reversals and substantial overlap in the induced optimization signals. Large shifts also concentrate on highly substitutable surface-form tokens, whereas tokens carrying problem-specific reasoning content are less sensitive. These findings show that privileged shifts fail to provide reliable answer-aligned directions, while their magnitudes primarily reflect counterfactual sensitivity rather than token-level learning value. Based on these observations, we propose Counterfactual Sensitivity Credit Reallocation (CSCR), a simple extension of GRPO that reduces credit for highly sensitive tokens and renormalizes token-level advantages to preserve both the original credit budget and verifier-determined direction. On long-CoT mathematical reasoning benchmarks, CSCR consistently outperforms GRPO baseline with the same number of policy updates. Targeted ablations further corroborate our diagnosis: privilege-induced directions are unreliable, moderate downweighting is most effective, and stronger modulation destabilizes optimization.