GRPO、Dr. GRPO 和 DAPO 是對一個數的三種運算：群體標準差恆等式

摘要

訓練語言模型進行推理的三種最流行方法，看似是三種不同的技巧，實則不然。這三種方法都調整同一個數值：標準差，反映提示詞所抽樣答案之間的分歧程度。當這類模型接受訓練時，它會多次回答每個問題，並由自動檢查器標記每個答案的對錯。這些標記的標準差衡量了分歧的程度：當答案在對錯之間均勻分佈時最大，當所有答案一致時則為零。群體相對策略最佳化（GRPO）除以這個數值，正確執行的GRPO（Dr. GRPO）省略除法，而解耦裁剪與動態採樣策略最佳化（DAPO）則捨棄標準差為零的群體。每種方法都被當作獨立的解決方案提出，但本文證明了它們其實是一個調節旋鈕的三種設定。這個旋鈕並非無關緊要：對於對錯獎勵而言，分歧正好等於訓練更新的幅度，即群體標準差恆等式。一個存在分歧的群體教得最多，而一個意見一致的群體則學不到任何東西，因此保持沉默。同樣的結果也指出哪些問題應獲得最高權重，以及每個問題需要多少次嘗試。本文在一個大型真實難度數據集（Big-Math）以及一次受控的訓練運作中，確認了這一直覺。看似無害的正規化步驟，正是決定學習發生在哪裡以及強度如何的關鍵調節旋鈕。

English

Three of the most popular methods for training language models to reason look like three different tricks. They are not. All three adjust a single number: standard deviation, reflecting how much a prompt's sampled answers disagree. When such a model is trained, it answers each problem many times, and an automatic checker marks every answer right or wrong. The standard deviation of those marks measures the disagreement: largest when the answers split evenly between right and wrong, and zero when they all agree. Group Relative Policy Optimization (GRPO) divides by this number, GRPO Done Right (Dr. GRPO) drops the division, and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) discards the groups where it is zero. Each is presented as its own fix, yet this paper proves they are three settings of one dial. That dial is not cosmetic: for right-or-wrong rewards, the disagreement is exactly the size of the training update, the group-standard-deviation identity. A split group teaches the most, while a unanimous group teaches nothing and falls silent. The same result says which problems deserve the most weight and how many tries each one needs. This paper confirms the intuition on a large real difficulty dataset (Big-Math) and in a controlled training run. What looks like a harmless normalization step is the dial that decides where learning happens and how strongly.