大型语言模型是所有字符的叠加：通过自对齐实现任意角色扮演

Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment

January 23, 2024

作者: Keming Lu, Bowen Yu, Chang Zhou, Jingren Zhou

cs.AI

摘要

为了增强开源大型语言模型（LLMs）在角色扮演方面的熟练度，人们已经付出了相当大的努力，试图模拟专有对应物。然而，我们认为LLMs天生具备角色扮演能力，这是因为它们在庞大的训练语料库中蕴含着对角色和潜在对话的广泛知识。因此，在这项研究中，我们介绍了Ditto，这是一种用于角色扮演的自我对齐方法。Ditto利用角色知识，鼓励一个遵循指令的LLM模拟角色扮演对话，作为阅读理解的一种变体。该方法创建了一个包含4,000个角色的角色扮演训练集，其规模超过了当前可用数据集的十倍。随后，我们使用这个自动生成的数据集对LLM进行微调，以增强其角色扮演能力。在评估我们精心构建且可重现的角色扮演基准测试和MT-Bench的角色扮演子集时，Ditto在各种参数规模下始终保持一致的角色身份，并在多轮角色扮演对话中提供准确的角色特定知识。值得注意的是，它胜过了所有开源角色扮演基准线，展现出与先进专有聊天机器人相媲美的性能水平。此外，我们展示了在角色扮演领域的首个全面交叉监督对齐实验，揭示了LLMs的内在能力将知识限制在角色扮演中。与此同时，角色扮演风格可以在较小模型的指导下轻松获得。我们在https://github.com/OFA-Sys/Ditto 开源了相关资源。

English

Considerable efforts have been invested in augmenting the role-playing proficiency of open-source large language models (LLMs) by emulating proprietary counterparts. Nevertheless, we posit that LLMs inherently harbor role-play capabilities, owing to the extensive knowledge of characters and potential dialogues ingrained in their vast training corpora. Thus, in this study, we introduce Ditto, a self-alignment method for role-play. Ditto capitalizes on character knowledge, encouraging an instruction-following LLM to simulate role-play dialogues as a variant of reading comprehension. This method creates a role-play training set comprising 4,000 characters, surpassing the scale of currently available datasets by tenfold regarding the number of roles. Subsequently, we fine-tune the LLM using this self-generated dataset to augment its role-playing capabilities. Upon evaluating our meticulously constructed and reproducible role-play benchmark and the roleplay subset of MT-Bench, Ditto, in various parameter scales, consistently maintains a consistent role identity and provides accurate role-specific knowledge in multi-turn role-play conversations. Notably, it outperforms all open-source role-play baselines, showcasing performance levels comparable to advanced proprietary chatbots. Furthermore, we present the first comprehensive cross-supervision alignment experiment in the role-play domain, revealing that the intrinsic capabilities of LLMs confine the knowledge within role-play. Meanwhile, the role-play styles can be easily acquired with the guidance of smaller models. We open-source related resources at https://github.com/OFA-Sys/Ditto.