使用扩散模型和引导梯度进行可控音乐制作

Controllable Music Production with Diffusion Models and Guidance Gradients

November 1, 2023

作者: Mark Levy, Bruno Di Giorgi, Floris Weers, Angelos Katharopoulos, Tom Nickson

cs.AI

摘要

我们展示了如何利用扩散模型中的条件生成来解决在制作44.1kHz立体声音频中具有采样时间指导的多种现实任务。我们考虑的场景包括音乐音频的延续、修复和再生，创建两个不同音乐曲目之间的平滑过渡，以及将期望的风格特征转移到现有音频片段中。我们通过在采样时间应用指导，使用一个简单的框架来实现这一点，该框架支持重建和分类损失，或两者的任何组合。这种方法确保生成的音频能够与其周围环境匹配，或者符合相对于任何适当的预训练分类器或嵌入模型指定的类分布或潜在表示。

English

We demonstrate how conditional generation from diffusion models can be used to tackle a variety of realistic tasks in the production of music in 44.1kHz stereo audio with sampling-time guidance. The scenarios we consider include continuation, inpainting and regeneration of musical audio, the creation of smooth transitions between two different music tracks, and the transfer of desired stylistic characteristics to existing audio clips. We achieve this by applying guidance at sampling time in a simple framework that supports both reconstruction and classification losses, or any combination of the two. This approach ensures that generated audio can match its surrounding context, or conform to a class distribution or latent representation specified relative to any suitable pre-trained classifier or embedding model.