FreeU：在Diffusion U-Net中的免费午餐

摘要

本文揭示了扩散 U-Net 的潜力，它被视为一种“免费午餐”，可以显著提高实时生成的质量。我们最初研究了 U-Net 架构对去噪过程的关键贡献，并确定其主干主要有助于去噪，而其跳跃连接主要将高频特征引入解码器模块，导致网络忽略主干语义。基于这一发现，我们提出了一种简单而有效的方法，称为“FreeU”，可以在不需要额外训练或微调的情况下增强生成质量。我们的关键洞察是有策略地重新加权源自 U-Net 跳跃连接和主干特征图的贡献，以利用 U-Net 架构的两个组成部分的优势。在图像和视频生成任务上取得的令人期待的结果表明，我们的 FreeU 可以轻松集成到现有的扩散模型中，例如 Stable Diffusion、DreamBooth、ModelScope、Rerender 和 ReVersion，只需几行代码即可提高生成质量。在推断过程中，您只需调整两个缩放因子。项目页面：https://chenyangsi.top/FreeU/。

English

In this paper, we uncover the untapped potential of diffusion U-Net, which serves as a "free lunch" that substantially improves the generation quality on the fly. We initially investigate the key contributions of the U-Net architecture to the denoising process and identify that its main backbone primarily contributes to denoising, whereas its skip connections mainly introduce high-frequency features into the decoder module, causing the network to overlook the backbone semantics. Capitalizing on this discovery, we propose a simple yet effective method-termed "FreeU" - that enhances generation quality without additional training or finetuning. Our key insight is to strategically re-weight the contributions sourced from the U-Net's skip connections and backbone feature maps, to leverage the strengths of both components of the U-Net architecture. Promising results on image and video generation tasks demonstrate that our FreeU can be readily integrated to existing diffusion models, e.g., Stable Diffusion, DreamBooth, ModelScope, Rerender and ReVersion, to improve the generation quality with only a few lines of code. All you need is to adjust two scaling factors during inference. Project page: https://chenyangsi.top/FreeU/.

FreeU：在Diffusion U-Net中的免费午餐

FreeU: Free Lunch in Diffusion U-Net

摘要

Support