AsyncDiff：通過異步去噪實現擴散模型的並行化

摘要

擴散模型因其在各種應用中具有強大的生成能力而引起了社區的廣泛興趣。然而，它們典型的多步驟序列去噪特性導致高累積延遲，因此排除了平行計算的可能性。為了解決這個問題，我們引入了AsyncDiff，這是一種通用且即插即用的加速方案，可以實現模型在多個設備之間的平行性。我們的方法將繁瑣的噪聲預測模型分為多個組件，並將每個組件分配給不同的設備。為了打破這些組件之間的依賴鏈，它將傳統的序列去噪轉換為一個非同步過程，通過利用連續擴散步驟中隱藏狀態之間的高相似性。因此，每個組件都可以在不同的設備上並行計算。所提出的策略顯著降低了推理延遲，同時對生成質量的影響最小。具體而言，對於Stable Diffusion v2.1，AsyncDiff實現了2.7倍的加速，幾乎沒有降低，並實現了4.0倍的加速，僅對CLIP分數有輕微的0.38降低，在四個NVIDIA A5000 GPU上。我們的實驗還表明，AsyncDiff可以輕鬆應用於具有令人鼓舞表現的視頻擴散模型。代碼可在https://github.com/czg1225/AsyncDiff找到。

English

Diffusion models have garnered significant interest from the community for their great generative ability across various applications. However, their typical multi-step sequential-denoising nature gives rise to high cumulative latency, thereby precluding the possibilities of parallel computation. To address this, we introduce AsyncDiff, a universal and plug-and-play acceleration scheme that enables model parallelism across multiple devices. Our approach divides the cumbersome noise prediction model into multiple components, assigning each to a different device. To break the dependency chain between these components, it transforms the conventional sequential denoising into an asynchronous process by exploiting the high similarity between hidden states in consecutive diffusion steps. Consequently, each component is facilitated to compute in parallel on separate devices. The proposed strategy significantly reduces inference latency while minimally impacting the generative quality. Specifically, for the Stable Diffusion v2.1, AsyncDiff achieves a 2.7x speedup with negligible degradation and a 4.0x speedup with only a slight reduction of 0.38 in CLIP Score, on four NVIDIA A5000 GPUs. Our experiments also demonstrate that AsyncDiff can be readily applied to video diffusion models with encouraging performances. The code is available at https://github.com/czg1225/AsyncDiff.

AsyncDiff：通過異步去噪實現擴散模型的並行化

AsyncDiff: Parallelizing Diffusion Models by Asynchronous Denoising

摘要

Summary

Support

Support