離散化獎勵模型

摘要

儘管獎勵模型被廣泛使用，但它們在塑造強化學習中的角色仍未被充分理解。獎勵模型提供了一個誘人的承諾：在沒有驗證者或人類評判者的情況下，它們能自動估算回應的品質。與通常產生二元分數的「可驗證獎勵」不同，獎勵模型通常產生連續分數，使其能夠對回應中的細微差異保持敏感。然而，我們表明這一看似優勢實則是一個嚴重弱點：許多流行的獎勵模型過度敏感，對同樣優秀的回應賦予不同分數。理論上，我們表明看似完美的獎勵模型可能極度過度敏感；實證上，這種過度敏感可能導致不良策略。我們提議以「辨別能力」和「特異性」（即過度敏感的補集）這兩個不同的度量來評估獎勵模型，取代現有的「獎勵模型準確性」概念。作為解決方案，我們描述了一種免訓練演算法，該演算法在任何神經獎勵模型上使用蒙地卡羅丟棄法以產生離散獎勵叢集。理論上，我們證明存在一些離散化方法，能以最小的辨別能力損失減少過度敏感；實證上，我們在受控與自然強化學習環境中皆表明，對獎勵進行離散化比直接在原始獎勵上訓練更能減少獎勵破解並產生更好的策略。

English

Despite their widespread use, the role of reward models in shaping reinforcement learning is poorly understood. Reward models offer a tempting promise: they automatically estimate response quality in the absence of verifiers or human judges. Unlike "verifiable rewards" which typically produce binary scores, reward models typically produce continuous scores, allowing them to be sensitive to fine-grained differences in responses. However, we show this apparent strength is a serious weakness: many popular reward models are oversensitive, assigning different scores to equally good responses. Theoretically, we show that seemingly perfect reward models can be highly oversensitive; empirically, this oversensitivity can lead to bad policies. In place of existing notions of "reward model accuracy," we propose evaluating reward models using distinct measures of "discriminative ability" and "specificity" (the complement of oversensitivity). As a solution, we describe a training-free algorithm that uses Monte Carlo dropout on any neural reward model to produce discrete reward clusters. Theoretically, we prove there exist discretizations that reduce oversensitivity at minimal expense of discriminative ability; empirically we show, in both controlled and natural RL settings, that discretizing rewards leads to less reward hacking and better policies than training on the original rewards.