推理時期的通用獎勵模型縮放

Inference-Time Scaling for Generalist Reward Modeling

April 3, 2025

作者: Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, Yu Wu

cs.AI

摘要

強化學習（RL）在大規模語言模型（LLMs）的後訓練中已被廣泛採用。近期，通過RL激勵LLMs的推理能力表明，適當的學習方法能夠實現有效的推理時可擴展性。RL的一個關鍵挑戰在於如何為LLMs在多種領域中獲取準確的獎勵信號，這些領域超越了可驗證問題或人工規則。在本研究中，我們探討了如何通過增加推理計算來改進通用查詢的獎勵建模（RM），即通用RM的推理時可擴展性，並進一步探討如何通過適當的學習方法提升性能-計算擴展的有效性。對於RM方法，我們採用了點對點生成式獎勵建模（GRM），以實現對不同輸入類型的靈活性及推理時擴展的潛力。在學習方法上，我們提出了自我原則批判微調（SPCT），通過在線RL培養GRM中的可擴展獎勵生成行為，自適應地生成原則並準確地進行批判，從而產生了DeepSeek-GRM模型。此外，為了實現有效的推理時擴展，我們使用並行採樣來擴展計算使用，並引入元RM來指導投票過程，以獲得更好的擴展性能。實證結果顯示，SPCT顯著提升了GRM的質量和可擴展性，在各種RM基準測試中超越了現有方法和模型，且未出現嚴重偏差，相比訓練時擴展能取得更優性能。DeepSeek-GRM在某些任務中仍面臨挑戰，我們相信這些挑戰可以通過未來在通用獎勵系統上的努力得到解決。這些模型將被發布並開源。

English

Reinforcement learning (RL) has been widely adopted in post-training for large language models (LLMs) at scale. Recently, the incentivization of reasoning capabilities in LLMs from RL indicates that proper learning methods could enable effective inference-time scalability. A key challenge of RL is to obtain accurate reward signals for LLMs in various domains beyond verifiable questions or artificial rules. In this work, we investigate how to improve reward modeling (RM) with more inference compute for general queries, i.e. the inference-time scalability of generalist RM, and further, how to improve the effectiveness of performance-compute scaling with proper learning methods. For the RM approach, we adopt pointwise generative reward modeling (GRM) to enable flexibility for different input types and potential for inference-time scaling. For the learning method, we propose Self-Principled Critique Tuning (SPCT) to foster scalable reward generation behaviors in GRMs through online RL, to generate principles adaptively and critiques accurately, resulting in DeepSeek-GRM models. Furthermore, for effective inference-time scaling, we use parallel sampling to expand compute usage, and introduce a meta RM to guide voting process for better scaling performance. Empirically, we show that SPCT significantly improves the quality and scalability of GRMs, outperforming existing methods and models in various RM benchmarks without severe biases, and could achieve better performance compared to training-time scaling. DeepSeek-GRM still meets challenges in some tasks, which we believe can be addressed by future efforts in generalist reward systems. The models will be released and open-sourced.