SemantiCodec：一種針對一般聲音的超低比特率語義音頻編解碼器

SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Sound

April 30, 2024

作者: Haohe Liu, Xuenan Xu, Yi Yuan, Mengyue Wu, Wenwu Wang, Mark D. Plumbley

cs.AI

摘要

大型語言模型（LLMs）通過將音頻轉換為離散標記的音頻編解碼器，顯著推進了音頻處理，從而使語言建模技術應用於音頻數據成為可能。然而，傳統編解碼器通常在高比特率下運行，或者僅在狹窄領域（如語音）內運作，缺乏進行有效語言建模所需的語義線索。為應對這些挑戰，我們提出了SemantiCodec，一種新型編解碼器，旨在將音頻壓縮為每秒不到一百個標記，涵蓋語音、一般音頻和音樂等多樣音頻類型，而無需犧牲質量。SemantiCodec採用雙編碼器架構：一個使用自監督AudioMAE的語義編碼器，在廣泛音頻數據上使用k均值聚類進行離散化，以及一個聲學編碼器來捕捉其餘細節。語義和聲學編碼器的輸出用於通過基於擴散模型的解碼器重建音頻。SemantiCodec有三種變體，標記速率分別為每秒25、50和100個，支持0.31 kbps至1.43 kbps之間的一系列超低比特率。實驗結果表明，SemantiCodec在重建質量上顯著優於最先進的Descript編解碼器。我們的結果還表明，即使在明顯較低的比特率下，SemantiCodec包含的語義信息也遠遠豐富於所有評估的音頻編解碼器。我們的代碼和演示可在https://haoheliu.github.io/SemantiCodec/上找到。

English

Large language models (LLMs) have significantly advanced audio processing through audio codecs that convert audio into discrete tokens, enabling the application of language modelling techniques to audio data. However, traditional codecs often operate at high bitrates or within narrow domains such as speech and lack the semantic clues required for efficient language modelling. Addressing these challenges, we introduce SemantiCodec, a novel codec designed to compress audio into fewer than a hundred tokens per second across diverse audio types, including speech, general audio, and music, without compromising quality. SemantiCodec features a dual-encoder architecture: a semantic encoder using a self-supervised AudioMAE, discretized using k-means clustering on extensive audio data, and an acoustic encoder to capture the remaining details. The semantic and acoustic encoder outputs are used to reconstruct audio via a diffusion-model-based decoder. SemantiCodec is presented in three variants with token rates of 25, 50, and 100 per second, supporting a range of ultra-low bit rates between 0.31 kbps and 1.43 kbps. Experimental results demonstrate that SemantiCodec significantly outperforms the state-of-the-art Descript codec on reconstruction quality. Our results also suggest that SemantiCodec contains significantly richer semantic information than all evaluated audio codecs, even at significantly lower bitrates. Our code and demos are available at https://haoheliu.github.io/SemantiCodec/.