小型語言模型作為基於評分標準之強化學習的評判者

摘要

基於評分量規的強化學習（Rubric-based reinforcement learning）透過針對實例特定標準對回應進行評分，將強化學習的應用擴展至具有精確答案或基於規則驗證器的任務之外。然而，這使得獎勵計算成本高昂：訓練過程需要重複的評分量規評判，通常依賴專有API或參數量達7B以上的本地生成式LLM評判器。我們研究較小的語言模型是否能作為高效且可靠的基於評分量規的評判器。為使此問題可量化衡量，我們建構了PointRubric與RaR-Science-Static兩個逐點式（pointwise）評分量規評估資料集，其中包含實例特定標準與逐項目滿足度標籤。我們比較了三種從小型模型中提取標準層級判斷的方法：生成式判決（Generative verdicts）、是/否Logprob邊際（Yes/No Logprob margins）以及探針式評判器（Probe judges）。在兩個資料集上，Qwen3-1.7B探針式評判器在這些方法中達到最強的標準層級一致性，優於生成式與Logprob評判器。作為GRPO獎勵模型使用時，它將策略在RaR-Science評分量規分數上從0.232訓練至0.643，而8B生成式評判器基線僅達到0.594，且基線所需的獎勵評判時間為前者的10.7倍。任務與領域遷移實驗進一步表明，探針式評判器在不同設定下能保留標準層級的獎勵結構。

English

Rubric-based reinforcement learning extends RL beyond tasks with exact answers or rule-based verifiers by scoring responses against instance-specific criteria. However, this makes reward computation expensive: training requires repeated rubric judging, often with proprietary APIs or local generative LLM judges with 7B parameters or more. We study whether smaller language models can serve as efficient and reliable rubric-based judges. To make this question measurable, we construct PointRubric and RaR-Science-Static, two pointwise rubric-based evaluation datasets with instance-specific criteria and itemwise satisfaction labels. We compare three ways of extracting criterion-level judgments from small models: Generative verdicts, Yes/No Logprob margins, and Probe judges. Across both datasets, the Qwen3-1.7B Probe judge achieves the strongest criterion-level agreement among these methods, outperforming Generative and Logprob judges. Used as a GRPO reward model, it trains a policy from 0.232 to 0.643 on RaR-Science rubric score, compared with 0.594 for an 8B Generative judge baseline, while the baseline requires 10.7times more reward-judge time. Task and domain transfer experiments further suggest that Probe judges preserve criterion-level reward structure across settings.