使用多智能體強化學習訓練社交推理語言模型

Training Language Models for Social Deduction with Multi-Agent Reinforcement Learning

February 9, 2025

作者: Bidipta Sarkar, Warren Xia, C. Karen Liu, Dorsa Sadigh

cs.AI

摘要

在多智能體環境中，以自然語言進行溝通是一個強大的工具，因為它使獨立智能體能夠在部分可觀察的情況下分享信息，並實現與人類的零-shot協調。然而，大多數先前的研究受限於依賴大量人類示範進行訓練，或缺乏生成自然且有用的溝通策略的能力。在這項研究中，我們訓練語言模型以自然語言就環境進行有產出的討論，而無需任何人類示範。我們將溝通問題分解為聆聽和說話兩部分。我們的關鍵想法是利用智能體的目標來預測有關世界的有用信息，作為引導溝通的密集獎勵信號。具體來說，我們通過訓練模型根據討論來預測環境信息，從而提高模型的聆聽技能，同時通過多智能體強化學習來改進模型的說話技能，通過根據其對其他智能體的影響來獎勵消息。為了研究在複雜社交環境中溝通的角色和必要性，我們研究了一個基於《Among Us》的具體社交推理遊戲，其中需要回答的關鍵問題是對手的身份。我們分析了由於我們的技術而出現的新行為，例如指控嫌疑人和提供證據，並發現這使得強有力的討論成為可能，勝率比標準RL提高了一倍。我們在https://socialdeductionllm.github.io/釋出了我們的代碼和模型。

English

Communicating in natural language is a powerful tool in multi-agent settings, as it enables independent agents to share information in partially observable settings and allows zero-shot coordination with humans. However, most prior works are limited as they either rely on training with large amounts of human demonstrations or lack the ability to generate natural and useful communication strategies. In this work, we train language models to have productive discussions about their environment in natural language without any human demonstrations. We decompose the communication problem into listening and speaking. Our key idea is to leverage the agent's goal to predict useful information about the world as a dense reward signal that guides communication. Specifically, we improve a model's listening skills by training them to predict information about the environment based on discussions, and we simultaneously improve a model's speaking skills with multi-agent reinforcement learning by rewarding messages based on their influence on other agents. To investigate the role and necessity of communication in complex social settings, we study an embodied social deduction game based on Among Us, where the key question to answer is the identity of an adversarial imposter. We analyze emergent behaviors due to our technique, such as accusing suspects and providing evidence, and find that it enables strong discussions, doubling the win rates compared to standard RL. We release our code and models at https://socialdeductionllm.github.io/