歌曲创作者：基于歌词的通用歌曲生成

SongCreator: Lyrics-based Universal Song Generation

September 9, 2024

作者: Shun Lei, Yixuan Zhou, Boshi Tang, Max W. Y. Lam, Feng Liu, Hangyu Liu, Jingcheng Wu, Shiyin Kang, Zhiyong Wu, Helen Meng

cs.AI

摘要

音乐是人类文化不可或缺的一部分，体现了人类的智慧和创造力，其中歌曲是重要组成部分。虽然先前的研究探索了歌曲生成的各个方面，如歌唱声音、声乐作曲和乐器编曲等，但在给定歌词的情况下生成既有人声又有伴奏的歌曲仍然是一个重大挑战，阻碍了音乐生成模型在现实世界中的应用。基于这一考虑，我们提出了SongCreator，这是一个旨在解决这一挑战的歌曲生成系统。该模型具有两个新颖设计：精心设计的双序列语言模型（DSLM）用于捕捉歌曲生成的人声和伴奏信息，以及DSL模型的额外注意力掩模策略，使我们的模型能够理解、生成和编辑歌曲，适用于各种与歌曲相关的生成任务。大量实验证明了SongCreator的有效性，通过在所有八项任务上取得了最先进或具有竞争力的表现。值得注意的是，在歌词到歌曲和歌词到人声方面，它大幅超越了先前的研究。此外，它能够通过不同提示独立控制生成歌曲中人声和伴奏的声学条件，展示了其潜在的适用性。我们的样本可在https://songcreator.github.io/ 上获取。

English

Music is an integral part of human culture, embodying human intelligence and creativity, of which songs compose an essential part. While various aspects of song generation have been explored by previous works, such as singing voice, vocal composition and instrumental arrangement, etc., generating songs with both vocals and accompaniment given lyrics remains a significant challenge, hindering the application of music generation models in the real world. In this light, we propose SongCreator, a song-generation system designed to tackle this challenge. The model features two novel designs: a meticulously designed dual-sequence language model (DSLM) to capture the information of vocals and accompaniment for song generation, and an additional attention mask strategy for DSLM, which allows our model to understand, generate and edit songs, making it suitable for various song-related generation tasks. Extensive experiments demonstrate the effectiveness of SongCreator by achieving state-of-the-art or competitive performances on all eight tasks. Notably, it surpasses previous works by a large margin in lyrics-to-song and lyrics-to-vocals. Additionally, it is able to independently control the acoustic conditions of the vocals and accompaniment in the generated song through different prompts, exhibiting its potential applicability. Our samples are available at https://songcreator.github.io/.