最佳實踐評論家優化

摘要

基於群組的強化學習方法（如GRPO）透過為每個提示採樣多個回應來避免訓練評論家（critic）。然而，一個可靠的評論家其實可以從單一回應中估計token層級的優勢，但標準的基於評論家的訓練方法往往不穩定。我們研究了這種不穩定性，並開發了**最佳實踐評論家優化（BPCO）**，這套方法結合了DPPO、受限於獎勵範圍的價值預測、蒙地卡羅價值目標、未正規化的策略優勢，以及長度自適應的廣義優勢估計（GAE）。由於評論家僅在訓練期間使用，BPCO也可以讓它依據獎勵定義資訊（例如參考答案或評分標準）作為條件，而這些資訊對策略是隱藏的。受控實驗分離了每個設計選擇的影響。在數學推理任務中，使用從1.5B參數到30B-A3B混合專家（MoE）的模型，BPCO一致地改進了一個強大的基於評論家的基線，並在每個提示僅採樣一個回應的情況下，匹配或超越了基於群組的基線。相同的方法也改善了基於評分標準獎勵的學習。這些結果顯示，精心設計的評論家為群組相對優勢估計提供了可靠的替代方案。程式碼可在 https://github.com/QPHutu/golden_critic 取得。

English

Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable. We study this instability and develop **Best Practice Critic Optimization (BPCO)**, a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy. Controlled experiments isolate the effect of each design choice. Across mathematical reasoning tasks with models ranging from 1.5B parameters to 30B-A3B mixtures of experts, BPCO improves a strong critic-based baseline consistently, and matches or exceeds a group-based baseline while sampling one response per prompt. The same recipe also improves learning with rubric-based rewards. These results show that a carefully designed critic provides a reliable alternative to group-relative advantage estimation. Code is available at https://github.com/QPHutu/golden_critic.