通过大型语言模型实现超人类语音理解的路线图

Roadmap towards Superhuman Speech Understanding using Large Language Models

October 17, 2024

作者: Fan Bu, Yuhao Zhang, Xidong Wang, Benyou Wang, Qun Liu, Haizhou Li

cs.AI

摘要

大型语言模型（LLMs）的成功促使了整合语音和音频数据的努力，旨在创建能够处理文本和非文本输入的通用基础模型。最近的进展，如GPT-4o，突显了端到端语音LLMs的潜力，可以保留非语义信息和世界知识，以实现更深层次的语音理解。为了指导语音LLMs的发展，我们提出了一个包括从基本自动语音识别（ASR）到能够将非语义信息与抽象声学知识整合用于复杂任务的先进超人模型在内的五级路线图。此外，我们设计了一个名为SAGI Benchmark的基准，标准化了这五个级别中各种任务的关键方面，揭示了使用抽象声学知识和能力完整性方面的挑战。我们的研究结果揭示了处理语音附加语线索和抽象声学知识方面存在的差距，并提出了未来的发展方向。本文概述了推进语音LLMs的路线图，介绍了一个用于评估的基准，并提供了关于它们当前的局限性和潜力的关键见解。

English

The success of large language models (LLMs) has prompted efforts to integrate speech and audio data, aiming to create general foundation models capable of processing both textual and non-textual inputs. Recent advances, such as GPT-4o, highlight the potential for end-to-end speech LLMs, which preserves non-semantic information and world knowledge for deeper speech understanding. To guide the development of speech LLMs, we propose a five-level roadmap, ranging from basic automatic speech recognition (ASR) to advanced superhuman models capable of integrating non-semantic information with abstract acoustic knowledge for complex tasks. Moreover, we design a benchmark, SAGI Bechmark, that standardizes critical aspects across various tasks in these five levels, uncovering challenges in using abstract acoustic knowledge and completeness of capability. Our findings reveal gaps in handling paralinguistic cues and abstract acoustic knowledge, and we offer future directions. This paper outlines a roadmap for advancing speech LLMs, introduces a benchmark for evaluation, and provides key insights into their current limitations and potential.