最佳实践评论家优化

摘要

诸如GRPO等针对大语言模型的基于组的强化学习方法，通过为每个提示采样多个响应来避免训练评论器。而一个可靠的评论器则可以从单个响应中估计词元级优势，但标准的基于评论器的训练方案往往不稳定。我们研究了这种不稳定性，并提出了**最佳实践评论器优化（BPCO）**，该方案结合了DPPO、奖励范围有界的价值预测、蒙特卡洛价值目标、未归一化的策略优势以及长度自适应的广义优势估计。由于评论器仅在训练期间使用，BPCO还可以让其以定义奖励的信息为条件，例如参考答案或评分标准，而这些信息对策略是隐藏的。受控实验单独评估了每个设计选择的影响。在从1.5B参数到30B-A3B混合专家模型的数学推理任务上，BPCO一致地改善了强基于评论器的基线，并且在每个提示仅采样一个响应的情况下，达到或超过了基于组的基线。同一方案也改进了使用基于评分标准的奖励时的学习效果。这些结果表明，精心设计的评论器为组相对优势估计提供了一种可靠的替代方案。代码可在 https://github.com/QPHutu/golden_critic 获取。

English

Group-based reinforcement learning methods such as GRPO for large language models avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable. We study this instability and develop **Best Practice Critic Optimization (BPCO)**, a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy. Controlled experiments isolate the effect of each design choice. Across mathematical reasoning tasks with models ranging from 1.5B parameters to 30B-A3B mixtures of experts, BPCO improves a strong critic-based baseline consistently, and matches or exceeds a group-based baseline while sampling one response per prompt. The same recipe also improves learning with rubric-based rewards. These results show that a carefully designed critic provides a reliable alternative to group-relative advantage estimation. Code is available at https://github.com/QPHutu/golden_critic.