语言模型可以在说话的同时倾听。

Language Model Can Listen While Speaking

August 5, 2024

作者: Ziyang Ma, Yakun Song, Chenpeng Du, Jian Cong, Zhuo Chen, Yuping Wang, Yuxuan Wang, Xie Chen

cs.AI

摘要

对话是人机交互（HCI）中最自然的方式。最近语音语言模型（SLM）的进展显著增强了基于语音的对话人工智能。然而，这些模型局限于基于轮次的对话，缺乏在实时口语场景中与人类互动的能力，例如在生成内容不理想时被打断。为解决这些限制，我们探索了交互式语音语言模型（iSLM）中的全双工建模（FDM），重点是增强实时互动，更明确地探索打断的基本能力。我们引入了一种新颖的模型设计，即听说语言模型（LSLM），这是一个端到端系统，配备了听和说两个通道。我们的LSLM采用基于标记的仅解码器TTS进行语音生成，以及用于实时音频输入的流式自监督学习（SSL）编码器。LSLM融合了两个通道以进行自回归生成，并实时检测轮次交替。我们探索了三种融合策略——早期融合、中间融合和晚期融合，其中中间融合在语音生成和实时互动之间实现了最佳平衡。两个实验设置，基于命令的FDM和基于语音的FDM，展示了LSLM对噪声的稳健性和对多样指令的敏感性。我们的结果突显了LSLM在最小影响现有系统的情况下实现全双工通信的能力。本研究旨在推动交互式语音对话系统的发展，增强其在现实世界环境中的适用性。

English

Dialogue serves as the most natural manner of human-computer interaction (HCI). Recent advancements in speech language models (SLM) have significantly enhanced speech-based conversational AI. However, these models are limited to turn-based conversation, lacking the ability to interact with humans in real-time spoken scenarios, for example, being interrupted when the generated content is not satisfactory. To address these limitations, we explore full duplex modeling (FDM) in interactive speech language models (iSLM), focusing on enhancing real-time interaction and, more explicitly, exploring the quintessential ability of interruption. We introduce a novel model design, namely listening-while-speaking language model (LSLM), an end-to-end system equipped with both listening and speaking channels. Our LSLM employs a token-based decoder-only TTS for speech generation and a streaming self-supervised learning (SSL) encoder for real-time audio input. LSLM fuses both channels for autoregressive generation and detects turn-taking in real time. Three fusion strategies -- early fusion, middle fusion, and late fusion -- are explored, with middle fusion achieving an optimal balance between speech generation and real-time interaction. Two experimental settings, command-based FDM and voice-based FDM, demonstrate LSLM's robustness to noise and sensitivity to diverse instructions. Our results highlight LSLM's capability to achieve duplex communication with minimal impact on existing systems. This study aims to advance the development of interactive speech dialogue systems, enhancing their applicability in real-world contexts.