小语言模型充当基于量规的强化学习的评判器

摘要

基于评分标准的强化学习通过依据实例特定的准则对回答进行评分，将强化学习扩展到了具有精确答案或基于规则的验证器的任务之外。然而，这使得奖励计算变得昂贵：训练过程需要反复进行评分标准的评判，通常依赖于专有API或参数量达70亿及以上的本地生成式大语言模型评判器。我们研究较小的语言模型是否能够作为高效且可靠的基于评分标准的评判器。为了使这一问题具有可度量性，我们构建了PointRubric和RaR-Science-Static这两个逐点评分式评估数据集，它们包含实例特定的准则和逐项满意度标签。我们比较了从小模型中提取准则级判断的三种方式：生成式裁决、是/否对数概率差值以及探针评判器。在两个数据集中，Qwen3-1.7B探针评判器在这些方法中取得了最强的准则级一致性，优于生成式和对数概率评判器。当将其用作GRPO奖励模型时，它使策略在RaR-Science评分标准上的得分从0.232提升至0.643，而8B生成式评判器基线为0.594，同时基线所需的奖励评判时间是其10.7倍。任务和领域迁移实验进一步表明，探针评判器在不同设置中能够保留准则级的奖励结构。

English

Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific criteria. However, this makes reward computation expensive: training requires repeated rubric judging, often with proprietary APIs or local generative LLM judges with 7B parameters or more. We study whether smaller language models can serve as efficient and reliable rubric-based judges. To make this question measurable, we construct PointRubric and RaR-Science-Static, two pointwise rubric-based evaluation datasets with instance-specific criteria and itemwise satisfaction labels. We compare three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges. Across both datasets, the Qwen3-1.7B Probe judge achieves the strongest criterion-level agreement among these methods, outperforming Generative and Logprob judges. Used as a GRPO reward model, it trains a policy from 0.232 to 0.643 on RaR-Science rubric score, compared with 0.594 for an 8B Generative judge baseline, while the baseline requires 10.7times more reward-judge time. Task and domain transfer experiments further suggest that Probe judges preserve criterion-level reward structure across settings.