SemantiCodec：一种面向一般声音的超低比特率语义音频编解码器

SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound

April 30, 2024

作者: Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, Mark D. Plumbley

cs.AI

摘要

大型语言模型（LLMs）通过将音频转换为离散标记的音频编解码器，显著推进了音频处理，从而使语言建模技术能够应用于音频数据。然而，传统编解码器通常以高比特率运行，或者仅限于狭窄领域，如语音，缺乏进行高效语言建模所需的语义线索。为了解决这些挑战，我们引入了SemantiCodec，这是一种新型编解码器，旨在将音频压缩为每秒不到一百个标记，涵盖语音、一般音频和音乐等多种音频类型，而不会降低质量。SemantiCodec采用双编码器架构：一个使用自监督的AudioMAE进行语义编码器，通过对大量音频数据进行k均值聚类进行离散化，以及一个声学编码器来捕获剩余细节。语义和声学编码器的输出用于通过基于扩散模型的解码器重建音频。SemantiCodec有三种变体，每秒的标记率分别为25、50和100，支持0.31 kbps至1.43 kbps之间的一系列超低比特率。实验结果表明，SemantiCodec在重建质量上明显优于最先进的Descript编解码器。我们的结果还表明，即使在明显更低的比特率下，SemantiCodec包含的语义信息显著丰富于所有评估过的音频编解码器。我们的代码和演示可在https://haoheliu.github.io/SemantiCodec/找到。

English

Large language models (LLMs) have significantly advanced audio processing through audio codecs that convert audio into discrete tokens, enabling the application of language modelling techniques to audio data. However, traditional codecs often operate at high bitrates or within narrow domains such as speech and lack the semantic clues required for efficient language modelling. Addressing these challenges, we introduce SemantiCodec, a novel codec designed to compress audio into fewer than a hundred tokens per second across diverse audio types, including speech, general audio, and music, without compromising quality. SemantiCodec features a dual-encoder architecture: a semantic encoder using a self-supervised AudioMAE, discretized using k-means clustering on extensive audio data, and an acoustic encoder to capture the remaining details. The semantic and acoustic encoder outputs are used to reconstruct audio via a diffusion-model-based decoder. SemantiCodec is presented in three variants with token rates of 25, 50, and 100 per second, supporting a range of ultra-low bit rates between 0.31 kbps and 1.43 kbps. Experimental results demonstrate that SemantiCodec significantly outperforms the state-of-the-art Descript codec on reconstruction quality. Our results also suggest that SemantiCodec contains significantly richer semantic information than all evaluated audio codecs, even at significantly lower bitrates. Our code and demos are available at https://haoheliu.github.io/SemantiCodec/.