DelTA: 基于可验证奖励的强化学习中的判别性令牌信用分配

摘要

基于可验证奖励的强化学习（RLVR）已成为提升大语言模型推理能力的核心方法。尽管其效果显著，但响应级奖励如何转化为词元级概率变化仍缺乏深入理解。我们提出RLVR更新的判别器视角，表明策略梯度更新方向隐式地充当词元梯度向量的线性判别器，从而决定学习过程中哪些词元概率得到提升或降低。在标准序列级RLVR下，该判别器由优势加权平均词元梯度向量形成的正负侧质心构建。然而，这种质心构建可能被共享的高频模式（如格式词元）主导，从而稀释了那些能更好区分高奖励与低奖励响应的稀疏判别方向。为解决这一局限，我们提出DelTA——一种判别性词元信用分配方法，通过估计词元系数来放大侧特异性词元梯度方向，同时降低共享或弱判别方向的权重。这些系数重新加权一个自归一化的RLVR替代目标，使有效的侧向质心更具对比性，从而重塑RLVR更新方向。在七个数学基准测试中，DelTA在Qwen3-8B-Base和Qwen3-14B-Base上分别比同规模最强基线高出3.26和2.62个平均分。代码生成、不同主干网络以及领域外评估的额外结果进一步证明了DelTA的泛化能力。

English

Reinforcement learning from verifiable rewards (RLVR) has emerged as a central technique for improving the reasoning capabilities of large language models. Despite its effectiveness, how response-level rewards translate into token-level probability changes remains poorly understood. We introduce a discriminator view of RLVR updates, showing that the policy-gradient update direction implicitly acts as a linear discriminator over token-gradient vectors and thereby determines which token probabilities are increased or decreased during learning. Under standard sequence-level RLVR, this discriminator is constructed from positive- and negative-side centroids formed by advantage-weighted averaging of token-gradient vectors. However, such centroid construction can be dominated by shared high-frequency patterns, such as formatting tokens, diluting sparse yet discriminative directions that better distinguish high-reward responses from low-reward ones. To address this limitation, we propose DelTA, a discriminative token credit assignment method that estimates token coefficients to amplify side-specific token-gradient directions and downweight shared or weakly discriminative ones. These coefficients reweight a self-normalized RLVR surrogate, making the effective side-wise centroids more contrastive and thereby reshaping the RLVR update direction. On seven mathematical benchmarks, DelTA outperforms the strongest same-scale baselines by 3.26 and 2.62 average points on Qwen3-8B-Base and Qwen3-14B-Base, respectively. Additional results on code generation, a different backbone, and out-of-domain evaluations further demonstrate the generalization ability of DelTA.