JEN-1 DreamStyler：通过关键参数调整实现定制音乐概念学习

JEN-1 DreamStyler: Customized Musical Concept Learning via Pivotal Parameters Tuning

June 18, 2024

作者: Boyu Chen, Peike Li, Yao Yao, Alex Wang

cs.AI

摘要

大型文本生成音乐模型取得了显著进展，促进了从提供的文本提示生成高质量且多样化的音乐作品。然而，输入的文本提示可能无法准确捕捉用户需求，特别是当目标是生成体现自指定参考集合中的特定概念的音乐时。在本文中，我们提出了一种新颖的定制文本生成音乐方法，可以从两分钟的参考音乐中捕捉概念并生成符合该概念的新音乐作品。我们通过使用参考音乐对预训练的文本生成音乐模型进行微调来实现这一目标。然而，直接微调所有参数会导致过拟合问题。为了解决这个问题，我们提出了一个关键参数调整方法，使模型能够吸收新概念同时保留其原始生成能力。此外，当向预训练模型引入多个概念时，我们发现潜在的概念冲突。我们提出了一个概念增强策略来区分多个概念，使经过微调的模型能够同时生成包含单个或多个概念的音乐。由于我们是第一个研究定制音乐生成任务的团队，我们还为这一新任务引入了一个新数据集和评估协议。我们提出的Jen1-DreamStyler在定性和定量评估中均优于几个基线模型。演示将在https://www.jenmusic.ai/research#DreamStyler 上提供。

English

Large models for text-to-music generation have achieved significant progress, facilitating the creation of high-quality and varied musical compositions from provided text prompts. However, input text prompts may not precisely capture user requirements, particularly when the objective is to generate music that embodies a specific concept derived from a designated reference collection. In this paper, we propose a novel method for customized text-to-music generation, which can capture the concept from a two-minute reference music and generate a new piece of music conforming to the concept. We achieve this by fine-tuning a pretrained text-to-music model using the reference music. However, directly fine-tuning all parameters leads to overfitting issues. To address this problem, we propose a Pivotal Parameters Tuning method that enables the model to assimilate the new concept while preserving its original generative capabilities. Additionally, we identify a potential concept conflict when introducing multiple concepts into the pretrained model. We present a concept enhancement strategy to distinguish multiple concepts, enabling the fine-tuned model to generate music incorporating either individual or multiple concepts simultaneously. Since we are the first to work on the customized music generation task, we also introduce a new dataset and evaluation protocol for the new task. Our proposed Jen1-DreamStyler outperforms several baselines in both qualitative and quantitative evaluations. Demos will be available at https://www.jenmusic.ai/research#DreamStyler.