AnimateLCM：通过解耦一致性学习加速个性化扩散模型和适配器的动画化

AnimateLCM: Accelerating the Animation of Personalized Diffusion Models and Adapters with Decoupled Consistency Learning

February 1, 2024

作者: Fu-Yun Wang, Zhaoyang Huang, Xiaoyu Shi, Weikang Bian, Guanglu Song, Yu Liu, Hongsheng Li

cs.AI

摘要

视频扩散模型因其能够生成既连贯又高保真度的视频而备受关注。然而，迭代去噪过程使其计算密集且耗时，从而限制了其应用。受一致性模型（CM）的启发，该模型将预训练的图像扩散模型提炼出来，以加速采样并减少步骤，以及其在有条件图像生成上的成功扩展——潜在一致性模型（LCM），我们提出了AnimateLCM，可在最少步骤内实现高保真度视频生成。我们提出了一种分离的一致性学习策略，而非直接在原始视频数据集上进行一致性学习，该策略将图像生成先验和运动生成先验的提炼分开，从而提高了训练效率并增强了生成的视觉质量。此外，为了实现在稳定扩散社区中插拔式适配器的组合以实现各种功能（例如，ControlNet 用于可控生成），我们提出了一种有效策略，将现有适配器适应到我们提炼的文本条件视频一致性模型上，或者从头开始训练适配器而不影响采样速度。我们在基于图像条件的视频生成和基于布局条件的视频生成中验证了所提出的策略，均取得了最佳结果。实验结果验证了我们提出方法的有效性。代码和权重将被公开。更多详细信息请访问 https://github.com/G-U-N/AnimateLCM。

English

Video diffusion models has been gaining increasing attention for its ability to produce videos that are both coherent and of high fidelity. However, the iterative denoising process makes it computationally intensive and time-consuming, thus limiting its applications. Inspired by the Consistency Model (CM) that distills pretrained image diffusion models to accelerate the sampling with minimal steps and its successful extension Latent Consistency Model (LCM) on conditional image generation, we propose AnimateLCM, allowing for high-fidelity video generation within minimal steps. Instead of directly conducting consistency learning on the raw video dataset, we propose a decoupled consistency learning strategy that decouples the distillation of image generation priors and motion generation priors, which improves the training efficiency and enhance the generation visual quality. Additionally, to enable the combination of plug-and-play adapters in stable diffusion community to achieve various functions (e.g., ControlNet for controllable generation). we propose an efficient strategy to adapt existing adapters to our distilled text-conditioned video consistency model or train adapters from scratch without harming the sampling speed. We validate the proposed strategy in image-conditioned video generation and layout-conditioned video generation, all achieving top-performing results. Experimental results validate the effectiveness of our proposed method. Code and weights will be made public. More details are available at https://github.com/G-U-N/AnimateLCM.