任务：令牌下采样以有效生成高分辨率图像

ToDo: Token Downsampling for Efficient Generation of High-Resolution Images

February 21, 2024

作者: Ethan Smith, Nayan Saxena, Aninda Saha

cs.AI

摘要

注意机制对图像扩散模型至关重要，然而，它们的二次计算复杂度限制了我们能够在合理时间和内存约束下处理的图像大小。本文研究了在生成图像模型中密集注意力的重要性，这些模型通常包含冗余特征，使它们适用于更稀疏的注意力机制。我们提出了一种新颖的无需训练的方法 ToDo，依赖于关键和值标记的标记降采样，可加速稳定扩散推断，对于常见尺寸可提高最多2倍，对于高分辨率如2048x2048可提高最多4.5倍或更多。我们证明了我们的方法在平衡高效吞吐量和保真度方面优于先前的方法。

English

Attention mechanism has been crucial for image diffusion models, however, their quadratic computational complexity limits the sizes of images we can process within reasonable time and memory constraints. This paper investigates the importance of dense attention in generative image models, which often contain redundant features, making them suitable for sparser attention mechanisms. We propose a novel training-free method ToDo that relies on token downsampling of key and value tokens to accelerate Stable Diffusion inference by up to 2x for common sizes and up to 4.5x or more for high resolutions like 2048x2048. We demonstrate that our approach outperforms previous methods in balancing efficient throughput and fidelity.