GRPO、Dr. GRPO 和 DAPO 是对同一数字的三种运算：组标准差恒等式

摘要

训练语言模型进行推理的三种最流行方法，看似是三种不同的技巧，实则不然。三者均调整同一个数值：标准差，它反映一个问题提示下各采样答案的分歧程度。当训练此类模型时，每个问题会被回答多次，自动校验器标记每个答案的正确或错误。这些标记的标准差衡量分歧度：当答案在正确与错误之间均匀分布时最大，当所有答案一致时为零。组相对策略优化（GRPO）除以该数值，正确版GRPO（Dr. GRPO）去掉除法操作，而解耦裁剪与动态采样策略优化（DAPO）则丢弃标准差为零的组。每种方法都作为独立的解决方案被提出，但本文证明它们只是同一调节旋钮上的三个设定。这个旋钮并非表面功夫：对于正误型奖励，分歧度恰好等于训练更新的大小——即组标准差恒等式。分歧大的组教会模型最多，而意见一致的组则毫无贡献、归于沉默。这一结论同时指出了哪些问题最值得赋予权重，以及每个问题需要多少次尝试。本文在一个大型真实难度数据集（Big-Math）和一次受控训练实验中证实了这一直觉。看似无害的标准化步骤，实则是决定学习发生在何处、学习强度有多大的调节旋钮。

English

Three of the most popular methods for training language models to reason look like three different tricks. They are not. All three adjust a single number: standard deviation, reflecting how much a prompt's sampled answers disagree. When such a model is trained, it answers each problem many times, and an automatic checker marks every answer right or wrong. The standard deviation of those marks measures the disagreement: largest when the answers split evenly between right and wrong, and zero when they all agree. Group Relative Policy Optimization (GRPO) divides by this number, GRPO Done Right (Dr. GRPO) drops the division, and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) discards the groups where it is zero. Each is presented as its own fix, yet this paper proves they are three settings of one dial. That dial is not cosmetic: for right-or-wrong rewards, the disagreement is exactly the size of the training update, the group-standard-deviation identity. A split group teaches the most, while a unanimous group teaches nothing and falls silent. The same result says which problems deserve the most weight and how many tries each one needs. This paper confirms the intuition on a large real difficulty dataset (Big-Math) and in a controlled training run. What looks like a harmless normalization step is the dial that decides where learning happens and how strongly.