歌曲創作者：基於歌詞的通用歌曲生成

SongCreator: Lyrics-based Universal Song Generation

September 9, 2024

作者: Shun Lei, Yixuan Zhou, Boshi Tang, Max W. Y. Lam, Feng Liu, Hangyu Liu, Jingcheng Wu, Shiyin Kang, Zhiyong Wu, Helen Meng

cs.AI

摘要

音樂是人類文化不可或缺的一部分，體現了人類的智慧和創造力，其中歌曲佔據了重要地位。儘管先前的研究已探索了歌曲生成的各個方面，如歌唱聲音、聲樂作曲和樂器編排等，但在給定歌詞的情況下生成既有人聲又有伴奏的歌曲仍然是一個重大挑戰，阻礙了音樂生成模型在現實世界中的應用。基於這一情況，我們提出了SongCreator，這是一個旨在應對這一挑戰的歌曲生成系統。該模型具有兩個新穎的設計：精心設計的雙序列語言模型（DSLM）用於捕捉歌曲生成的人聲和伴奏信息，以及DSL模型的額外注意力遮罩策略，使我們的模型能夠理解、生成和編輯歌曲，適用於各種與歌曲相關的生成任務。大量實驗證明了SongCreator的有效性，通過在所有八個任務上實現了最先進或具有競爭力的表現。值得注意的是，在歌詞轉歌曲和歌詞轉人聲方面，它明顯超越了先前的作品。此外，通過不同提示，它能夠獨立控制生成歌曲中人聲和伴奏的聲學條件，展示了其潛在的應用價值。我們的樣本可在https://songcreator.github.io/ 上找到。

English

Music is an integral part of human culture, embodying human intelligence and creativity, of which songs compose an essential part. While various aspects of song generation have been explored by previous works, such as singing voice, vocal composition and instrumental arrangement, etc., generating songs with both vocals and accompaniment given lyrics remains a significant challenge, hindering the application of music generation models in the real world. In this light, we propose SongCreator, a song-generation system designed to tackle this challenge. The model features two novel designs: a meticulously designed dual-sequence language model (DSLM) to capture the information of vocals and accompaniment for song generation, and an additional attention mask strategy for DSLM, which allows our model to understand, generate and edit songs, making it suitable for various song-related generation tasks. Extensive experiments demonstrate the effectiveness of SongCreator by achieving state-of-the-art or competitive performances on all eight tasks. Notably, it surpasses previous works by a large margin in lyrics-to-song and lyrics-to-vocals. Additionally, it is able to independently control the acoustic conditions of the vocals and accompaniment in the generated song through different prompts, exhibiting its potential applicability. Our samples are available at https://songcreator.github.io/.