超越基于回合的游戏：利用双工模型实现实时对话

Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Models

June 22, 2024

作者: Xinrong Zhang, Yingfa Chen, Shengding Hu, Xu Han, Zihang Xu, Yuanwei Xu, Weilin Zhao, Maosong Sun, Zhiyuan Liu

cs.AI

摘要

随着大型语言模型（LLMs）日益渗透到日常生活中，对模拟人类对话的实时交互需求不断增长。传统的基于轮次的聊天系统由LLMs驱动，阻止用户在系统生成响应时进行口头交互。为了克服这些限制，我们将现有的LLMs调整为双工模型，使这些LLMs能够在生成输出的同时倾听用户，并动态调整以为用户提供即时反馈，例如对中断的响应。具体而言，我们将对话的查询和响应划分为多个时间片段，然后采用时分复用（TDM）编码-解码策略来伪同时处理这些片段。此外，为了使LLMs能够熟练处理实时对话，我们构建了一个微调数据集，其中包含交替的查询和响应时间片段，以及覆盖瞬时交互中典型反馈类型。我们的实验表明，尽管对话的查询和响应被划分为不完整的片段进行处理，但LLMs在我们的数据集上经过少量微调步骤后可以保持其在标准基准上的原始性能。自动化和人工评估表明，双工模型使用户与AI的交互更加自然和类人，与普通LLMs相比，大大提高了用户满意度。我们的双工模型和数据集将会发布。

English

As large language models (LLMs) increasingly permeate daily lives, there is a growing demand for real-time interactions that mirror human conversations. Traditional turn-based chat systems driven by LLMs prevent users from verbally interacting with the system while it is generating responses. To overcome these limitations, we adapt existing LLMs to duplex models so that these LLMs can listen for users while generating output and dynamically adjust themselves to provide users with instant feedback. % such as in response to interruptions. Specifically, we divide the queries and responses of conversations into several time slices and then adopt a time-division-multiplexing (TDM) encoding-decoding strategy to pseudo-simultaneously process these slices. Furthermore, to make LLMs proficient enough to handle real-time conversations, we build a fine-tuning dataset consisting of alternating time slices of queries and responses as well as covering typical feedback types in instantaneous interactions. Our experiments show that although the queries and responses of conversations are segmented into incomplete slices for processing, LLMs can preserve their original performance on standard benchmarks with a few fine-tuning steps on our dataset. Automatic and human evaluation indicate that duplex models make user-AI interactions more natural and human-like, and greatly improve user satisfaction compared to vanilla LLMs. Our duplex model and dataset will be released.