DeepCache：加速扩散模型的自由化

DeepCache: Accelerating Diffusion Models for Free

December 1, 2023

作者: Xinyin Ma, Gongfan Fang, Xinchao Wang

cs.AI

摘要

最近，由于其卓越的生成能力，扩散模型在图像合成领域引起了前所未有的关注。尽管这些模型强大，但往往会产生大量的计算成本，主要归因于顺序去噪过程和庞大的模型尺寸。传统的扩散模型压缩方法通常涉及大量的重新训练，带来了成本和可行性方面的挑战。本文介绍了DeepCache，一种新颖的无需训练的范例，从模型架构的角度加速扩散模型。DeepCache利用扩散模型顺序去噪步骤中观察到的固有时间冗余，缓存并检索相邻去噪阶段的特征，从而削减了冗余计算。利用U-Net的特性，我们以一种非常廉价的方式重复使用高级特征，同时更新低级特征。这一创新策略进而使Stable Diffusion v1.5的加速比因子达到2.3倍，仅在CLIP Score下降0.05的情况下，以及LDM-4-G的加速比因子达到4.1倍，在ImageNet上FID略微下降0.22。我们的实验还展示了DeepCache相对于现有的修剪和蒸馏方法的优越性，这些方法需要重新训练，并且与当前的采样技术兼容。此外，我们发现在相同的吞吐量下，DeepCache能够有效地实现与DDIM或PLMS相当甚至略有改进的结果。代码可在https://github.com/horseee/DeepCache找到。

English

Diffusion models have recently gained unprecedented attention in the field of image synthesis due to their remarkable generative capabilities. Notwithstanding their prowess, these models often incur substantial computational costs, primarily attributed to the sequential denoising process and cumbersome model size. Traditional methods for compressing diffusion models typically involve extensive retraining, presenting cost and feasibility challenges. In this paper, we introduce DeepCache, a novel training-free paradigm that accelerates diffusion models from the perspective of model architecture. DeepCache capitalizes on the inherent temporal redundancy observed in the sequential denoising steps of diffusion models, which caches and retrieves features across adjacent denoising stages, thereby curtailing redundant computations. Utilizing the property of the U-Net, we reuse the high-level features while updating the low-level features in a very cheap way. This innovative strategy, in turn, enables a speedup factor of 2.3times for Stable Diffusion v1.5 with only a 0.05 decline in CLIP Score, and 4.1times for LDM-4-G with a slight decrease of 0.22 in FID on ImageNet. Our experiments also demonstrate DeepCache's superiority over existing pruning and distillation methods that necessitate retraining and its compatibility with current sampling techniques. Furthermore, we find that under the same throughput, DeepCache effectively achieves comparable or even marginally improved results with DDIM or PLMS. The code is available at https://github.com/horseee/DeepCache