面向大语言模型强化学习的预测性散度掩码

摘要

强化学习在大语言模型中的应用通常依赖信任区域掩码来稳定离线策略更新。主流的PPO风格方法使用采样令牌的重要性比率作为两项判据：邻近性判据，用于判断策略是否偏离行为策略过远；方向性判据，用于判断更新是否进一步扩大这种偏离。近期研究DPPO通过将PPO基于比率的测试替换为行为策略与训练策略之间的概率散度，改进了邻近性判据。然而其方向性判据仍继承自PPO，仅当采样令牌的重要性比率偏离1时才会对令牌进行掩码处理。我们观察到这种基于比率的方向性判据是一种单样本代理，其符号可能与定义邻近性判据的散度变化产生冲突。因此我们提出预测性散度掩码，通过判断下一步策略梯度更新将增大还是减小信任区域所采用的同一散度值。针对LLM强化学习中使用的离散softmax策略，我们推导出了该预测的闭式解。由于生产环境的推理引擎仅暴露词汇表的截断视图（top-K），我们开发了两种轻量级top-K估计器用于该预测。详细分析表明，与采样比率相比，基于散度的方向判据与散度实际变化的一致性更优，由此生成的掩码能够跨模型规模与精度设置提升强化学习训练效果。

English

Reinforcement learning for large language models (LLMs) typically relies on trust-region masks to stabilize off-policy updates. The dominant PPO-style approach uses the sampled-token importance ratio for two criteria: a proximity criterion, which asks whether the policy has moved too far from the behavior policy, and a direction criterion, which asks whether the update pushes it farther away. Recent work DPPO improves the proximity criterion by replacing PPO's ratio-based test with a probability divergence between the behavior and training policies. However, its direction criterion is still inherited from PPO. A token can be masked only when the sampled-token importance ratio moves away from one. We observe that this ratio-based direction criterion is a single-sample proxy that can disagree in sign with the change of the divergence that defines the proximity criterion. We therefore propose the predictive divergence mask, which asks whether the next policy-gradient step will increase or decrease the same divergence used by the trust region. For the discrete softmax policies used in LLM RL, we derive this prediction in closed form. Because production rollout engines expose only a truncated (top-K) view of the vocabulary, we develop two lightweight top-K estimators for this prediction. Detailed analysis shows the divergence-based direction is better aligned with the realized change of the divergence than the sampled ratio, and the resulting masks improve RL training across model scales and precision settings.