Takin：一组优质的零-shot语音生成模型

Takin: A Cohort of Superior Quality Zero-shot Speech Generation Models

September 18, 2024

作者: EverestAI, Sijin Chen, Yuan Feng, Laipeng He, Tianwei He, Wendi He, Yanni Hu, Bin Lin, Yiting Lin, Pengfei Tan, Chengwei Tian, Chen Wang, Zhicheng Wang, Ruoye Xie, Jingjing Yin, Jianhao Ye, Jixun Yao, Quanlei Yan, Yuguang Yang

cs.AI

摘要

随着大数据和大型语言模型时代的到来，零-shot 个性化快速定制已经成为一个重要趋势。在本报告中，我们介绍了Takin AudioLLM，这是一系列主要包括 Takin TTS、Takin VC 和 Takin Morphing 等技术和模型，专为有声读物制作而设计。这些模型能够进行零-shot 语音生成，生成几乎无法与真实人类语音区分的高质量语音，并帮助个人根据自己的需求定制语音内容。具体来说，我们首先介绍了 Takin TTS，这是一个神经编解码器语言模型，基于增强型神经语音编解码器和多任务训练框架，能够以零-shot 方式生成高保真自然语音。对于 Takin VC，我们提倡一种有效的内容和音色联合建模方法来提高说话者相似度，同时提倡基于条件流匹配的解码器来进一步增强其自然性和表现力。最后，我们提出了 Takin Morphing 系统，采用高度解耦和先进的音色和韵律建模方法，使个人能够以精确可控的方式定制其偏好的音色和韵律进行语音生成。大量实验证实了我们的 Takin AudioLLM 系列模型的有效性和鲁棒性。有关详细演示，请参阅 https://takinaudiollm.github.io。

English

With the advent of the big data and large language model era, zero-shot personalized rapid customization has emerged as a significant trend. In this report, we introduce Takin AudioLLM, a series of techniques and models, mainly including Takin TTS, Takin VC, and Takin Morphing, specifically designed for audiobook production. These models are capable of zero-shot speech production, generating high-quality speech that is nearly indistinguishable from real human speech and facilitating individuals to customize the speech content according to their own needs. Specifically, we first introduce Takin TTS, a neural codec language model that builds upon an enhanced neural speech codec and a multi-task training framework, capable of generating high-fidelity natural speech in a zero-shot way. For Takin VC, we advocate an effective content and timbre joint modeling approach to improve the speaker similarity, while advocating for a conditional flow matching based decoder to further enhance its naturalness and expressiveness. Last, we propose the Takin Morphing system with highly decoupled and advanced timbre and prosody modeling approaches, which enables individuals to customize speech production with their preferred timbre and prosody in a precise and controllable manner. Extensive experiments validate the effectiveness and robustness of our Takin AudioLLM series models. For detailed demos, please refer to https://takinaudiollm.github.io.