SCEdit：通过跳跃连接编辑实现高效可控的图像扩散生成

SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing

December 18, 2023

作者: Zeyinzi Jiang, Chaojie Mao, Yulin Pan, Zhen Han, Jingfeng Zhang

cs.AI

摘要

图像扩散模型已被应用于各种任务，如文本到图像生成和可控图像合成。最近的研究引入了微调方法，对原始模型进行微小调整，取得了在基础生成扩散模型的特定改进方面的有希望的结果。我们不是修改扩散模型的主干，而是深入探讨了 U-Net 中跳跃连接的作用，并揭示了在编码器和解码器之间聚合远距离信息的分层特征对图像生成的内容和质量产生重大影响。基于这一观察，我们提出了一种高效的生成微调框架，名为 SCEdit，它集成和编辑跳跃连接，使用名为 SC-Tuner 的轻量级调节模块。此外，所提出的框架通过向 Controllable SC-Tuner 注入不同条件，实现了对可控图像合成的简化和统一网络设计，使其能够轻松扩展到多条件输入。我们的 SCEdit 大大减少了训练参数、内存使用和计算开销，因为其轻量级调节器，仅将反向传播传递到解码器块。在文本到图像生成和可控图像合成任务上进行的大量实验表明，我们的方法在效率和性能方面优越。项目页面：https://scedit.github.io/

English

Image diffusion models have been utilized in various tasks, such as text-to-image generation and controllable image synthesis. Recent research has introduced tuning methods that make subtle adjustments to the original models, yielding promising results in specific adaptations of foundational generative diffusion models. Rather than modifying the main backbone of the diffusion model, we delve into the role of skip connection in U-Net and reveal that hierarchical features aggregating long-distance information across encoder and decoder make a significant impact on the content and quality of image generation. Based on the observation, we propose an efficient generative tuning framework, dubbed SCEdit, which integrates and edits Skip Connection using a lightweight tuning module named SC-Tuner. Furthermore, the proposed framework allows for straightforward extension to controllable image synthesis by injecting different conditions with Controllable SC-Tuner, simplifying and unifying the network design for multi-condition inputs. Our SCEdit substantially reduces training parameters, memory usage, and computational expense due to its lightweight tuners, with backward propagation only passing to the decoder blocks. Extensive experiments conducted on text-to-image generation and controllable image synthesis tasks demonstrate the superiority of our method in terms of efficiency and performance. Project page: https://scedit.github.io/