VITA-Audio：面向高效大规模语音语言模型的快速交错跨模态令牌生成

VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient Large Speech-Language Model

May 6, 2025

作者: Zuwei Long, Yunhang Shen, Chaoyou Fu, Heting Gao, Lijiang Li, Peixian Chen, Mengdan Zhang, Hang Shao, Jian Li, Jinlong Peng, Haoyu Cao, Ke Li, Rongrong Ji, Xing Sun

cs.AI

摘要

随着对自然人机交互需求的日益增长，基于语音的系统因其作为日常交流中最常见的形式之一而受到越来越多的关注。然而，现有的语音模型在流式处理中生成首个音频标记时仍面临高延迟问题，这成为部署过程中的一大瓶颈。为解决此问题，我们提出了VITA-Audio，一种能够快速生成音频-文本标记的端到端大型语音模型。具体而言，我们引入了一个轻量级的多模态交叉标记预测（MCTP）模块，该模块在单次模型前向传播中高效生成多个音频标记，不仅加速了推理过程，还显著降低了流式场景下生成首个音频的延迟。此外，我们探索了一种四阶段渐进式训练策略，以在最小化语音质量损失的前提下实现模型加速。据我们所知，VITA-Audio是首个能够在首次前向传播中生成音频输出的多模态大语言模型，实现了具有极低延迟的实时对话能力。VITA-Audio完全可复现，且仅使用开源数据进行训练。实验结果表明，我们的模型在70亿参数规模下实现了3至5倍的推理加速，同时在自动语音识别（ASR）、文本转语音（TTS）及口语问答（SQA）任务的多项基准测试中，显著优于相似模型规模的开源模型。

English

With the growing requirement for natural human-computer interaction, speech-based systems receive increasing attention as speech is one of the most common forms of daily communication. However, the existing speech models still experience high latency when generating the first audio token during streaming, which poses a significant bottleneck for deployment. To address this issue, we propose VITA-Audio, an end-to-end large speech model with fast audio-text token generation. Specifically, we introduce a lightweight Multiple Cross-modal Token Prediction (MCTP) module that efficiently generates multiple audio tokens within a single model forward pass, which not only accelerates the inference but also significantly reduces the latency for generating the first audio in streaming scenarios. In addition, a four-stage progressive training strategy is explored to achieve model acceleration with minimal loss of speech quality. To our knowledge, VITA-Audio is the first multi-modal large language model capable of generating audio output during the first forward pass, enabling real-time conversational capabilities with minimal latency. VITA-Audio is fully reproducible and is trained on open-source data only. Experimental results demonstrate that our model achieves an inference speedup of 3~5x at the 7B parameter scale, but also significantly outperforms open-source models of similar model size on multiple benchmarks for automatic speech recognition (ASR), text-to-speech (TTS), and spoken question answering (SQA) tasks.