DelTA：基於可驗證獎勵的強化學習中的辨識性詞元信用分配

摘要

可驗證獎勵強化學習（RLVR）已成為提升大型語言模型推理能力的核心技術。儘管其效果顯著，但回應層級獎勵如何轉化為詞元層級機率變化，至今仍未被充分理解。我們引入了RLVR更新的鑑別器觀點，指出策略梯度更新方向隱含地充當了詞元梯度向量上的線性鑑別器，從而決定學習過程中哪些詞元的機率被增加或減少。在標準的序列層級RLVR下，此鑑別器由優勢加權平均詞元梯度向量所形成的正側與負側質心建構而成。然而，這種質心建構可能被共有的高頻模式（例如格式符號）主導，從而削弱了那些能更好區分高獎勵回應與低獎勵回應的稀疏但具鑑別性的方向。為了解決此限制，我們提出了DelTA，這是一種鑑別性詞元信用分配方法，通過估算詞元係數來放大特定側的詞元梯度方向，並降低共享或弱鑑別方向的權重。這些係數重新加權了自我正規化的RLVR替代函數，使有效的側向質心更具對比性，從而重塑RLVR更新方向。在七個數學基準測試上，DelTA在Qwen3-8B-Base和Qwen3-14B-Base上分別比最強的同規模基線高出3.26和2.62個平均點。在程式碼生成、不同骨幹模型及領域外評估上的額外結果，進一步證明了DelTA的泛化能力。

English

Reinforcement learning from verifiable rewards (RLVR) has emerged as a central technique for improving the reasoning capabilities of large language models. Despite its effectiveness, how response-level rewards translate into token-level probability changes remains poorly understood. We introduce a discriminator view of RLVR updates, showing that the policy-gradient update direction implicitly acts as a linear discriminator over token-gradient vectors and thereby determines which token probabilities are increased or decreased during learning. Under standard sequence-level RLVR, this discriminator is constructed from positive- and negative-side centroids formed by advantage-weighted averaging of token-gradient vectors. However, such centroid construction can be dominated by shared high-frequency patterns, such as formatting tokens, diluting sparse yet discriminative directions that better distinguish high-reward responses from low-reward ones. To address this limitation, we propose DelTA, a discriminative token credit assignment method that estimates token coefficients to amplify side-specific token-gradient directions and downweight shared or weakly discriminative ones. These coefficients reweight a self-normalized RLVR surrogate, making the effective side-wise centroids more contrastive and thereby reshaping the RLVR update direction. On seven mathematical benchmarks, DelTA outperforms the strongest same-scale baselines by 3.26 and 2.62 average points on Qwen3-8B-Base and Qwen3-14B-Base, respectively. Additional results on code generation, a different backbone, and out-of-domain evaluations further demonstrate the generalization ability of DelTA.