共識遊戲：通過均衡搜索進行語言模型生成

摘要

當應用於問答和其他文本生成任務時，語言模型（LMs）可以通過生成式查詢（從輸出分佈中抽樣答案）或歧視式查詢（使用它們對一組候選輸出進行評分或排名）。這些程序有時會產生非常不同的預測。我們如何調和互相不相容的評分程序，以獲得一致的LM預測？我們引入了一種新的、無需訓練的、博弈論程序，用於語言模型解碼。我們的方法將語言模型解碼視為一個正規化的不完全信息序列信號博弈 - 我們稱之為共識博弈 - 在這個過程中，生成器試圖使用自然語言句子向歧視器傳達一個抽象的正確性參數。我們開發了計算程序，用於找到這個博弈的近似均衡，從而產生一種我們稱之為均衡排名的解碼算法。應用於大量任務（包括閱讀理解、常識推理、數學問題解決和對話），均衡排名一致地，有時顯著地，優於現有的LM解碼程序 - 在多個基準測試中，我們觀察到將均衡排名應用於LLaMA-7B比使用更大的LLaMA-65B和PaLM-540B模型效果更好。這些結果突顯了博弈論工具在解決LM中的真實性和一致性等基本挑戰方面的潛力。

English

When applied to question answering and other text generation tasks, language models (LMs) may be queried generatively (by sampling answers from their output distribution) or discriminatively (by using them to score or rank a set of candidate outputs). These procedures sometimes yield very different predictions. How do we reconcile mutually incompatible scoring procedures to obtain coherent LM predictions? We introduce a new, a training-free, game-theoretic procedure for language model decoding. Our approach casts language model decoding as a regularized imperfect-information sequential signaling game - which we term the CONSENSUS GAME - in which a GENERATOR seeks to communicate an abstract correctness parameter using natural language sentences to a DISCRIMINATOR. We develop computational procedures for finding approximate equilibria of this game, resulting in a decoding algorithm we call EQUILIBRIUM-RANKING. Applied to a large number of tasks (including reading comprehension, commonsense reasoning, mathematical problem-solving, and dialog), EQUILIBRIUM-RANKING consistently, and sometimes substantially, improves performance over existing LM decoding procedures - on multiple benchmarks, we observe that applying EQUILIBRIUM-RANKING to LLaMA-7B outperforms the much larger LLaMA-65B and PaLM-540B models. These results highlight the promise of game-theoretic tools for addressing fundamental challenges of truthfulness and consistency in LMs.

共識遊戲：通過均衡搜索進行語言模型生成

The Consensus Game: Language Model Generation via Equilibrium Search

摘要

Support