大型語言模型是所有字符的超位置：通過自對齊實現任意角色扮演

Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment

January 23, 2024

作者: Keming Lu, Bowen Yu, Chang Zhou, Jingren Zhou

cs.AI

摘要

為了增強開源大型語言模型（LLMs）在角色扮演方面的能力，已經投入了相當多的努力，試圖模擬專有對應物。然而，我們認為LLMs本質上具有角色扮演能力，這是因為它們在龐大的訓練語料庫中擁有對角色和潛在對話的廣泛知識。因此，在這項研究中，我們介紹了Ditto，一種用於角色扮演的自我對齊方法。Ditto利用角色知識，鼓勵一個遵循指示的LLM模擬角色扮演對話，作為閱讀理解的一種變體。該方法創建了一個包含4,000個角色的角色扮演訓練集，其角色數量比當前可用數據集大十倍。隨後，我們使用這個自生成的數據集對LLM進行微調，以增強其角色扮演能力。通過評估我們精心構建且可重現的角色扮演基準和MT-Bench的角色扮演子集，Ditto在各種參數規模下始終保持一致的角色身份，並在多輪角色扮演對話中提供準確的角色特定知識。值得注意的是，它優於所有開源角色扮演基準，展現出與先進專有聊天機器人相當的性能水平。此外，我們提出了在角色扮演領域的第一個全面的交叉監督對齊實驗，揭示了LLMs的內在能力將知識限制在角色扮演之內。與此同時，角色扮演風格可以在較小模型的指導下輕鬆獲得。我們在https://github.com/OFA-Sys/Ditto 開源相關資源。

English

Considerable efforts have been invested in augmenting the role-playing proficiency of open-source large language models (LLMs) by emulating proprietary counterparts. Nevertheless, we posit that LLMs inherently harbor role-play capabilities, owing to the extensive knowledge of characters and potential dialogues ingrained in their vast training corpora. Thus, in this study, we introduce Ditto, a self-alignment method for role-play. Ditto capitalizes on character knowledge, encouraging an instruction-following LLM to simulate role-play dialogues as a variant of reading comprehension. This method creates a role-play training set comprising 4,000 characters, surpassing the scale of currently available datasets by tenfold regarding the number of roles. Subsequently, we fine-tune the LLM using this self-generated dataset to augment its role-playing capabilities. Upon evaluating our meticulously constructed and reproducible role-play benchmark and the roleplay subset of MT-Bench, Ditto, in various parameter scales, consistently maintains a consistent role identity and provides accurate role-specific knowledge in multi-turn role-play conversations. Notably, it outperforms all open-source role-play baselines, showcasing performance levels comparable to advanced proprietary chatbots. Furthermore, we present the first comprehensive cross-supervision alignment experiment in the role-play domain, revealing that the intrinsic capabilities of LLMs confine the knowledge within role-play. Meanwhile, the role-play styles can be easily acquired with the guidance of smaller models. We open-source related resources at https://github.com/OFA-Sys/Ditto.