FreeU：在Diffusion U-Net中的免費午餐

摘要

本文揭示了擴散 U-Net 的潛力，它被視為一種「免費午餐」，顯著提高了即時生成的質量。我們首先研究了 U-Net 結構對降噪過程的關鍵貢獻，發現其主幹主要貢獻於降噪，而其跳躍連接主要將高頻特徵引入解碼器模塊，導致網絡忽略主幹語義。基於這一發現，我們提出了一種簡單而有效的方法，稱為「FreeU」，它可以提高生成質量，無需額外的訓練或微調。我們的關鍵見解是從戰略上重新加權來自 U-Net 跳躍連接和主幹特徵圖的貢獻，以利用 U-Net 結構的兩個組成部分的優勢。在圖像和視頻生成任務上取得的令人期待的結果表明，我們的 FreeU 可輕鬆集成到現有的擴散模型中，例如 Stable Diffusion、DreamBooth、ModelScope、Rerender 和 ReVersion，只需幾行代碼即可提高生成質量。您只需在推理過程中調整兩個縮放因子即可。項目頁面：https://chenyangsi.top/FreeU/。

English

In this paper, we uncover the untapped potential of diffusion U-Net, which serves as a "free lunch" that substantially improves the generation quality on the fly. We initially investigate the key contributions of the U-Net architecture to the denoising process and identify that its main backbone primarily contributes to denoising, whereas its skip connections mainly introduce high-frequency features into the decoder module, causing the network to overlook the backbone semantics. Capitalizing on this discovery, we propose a simple yet effective method-termed "FreeU" - that enhances generation quality without additional training or finetuning. Our key insight is to strategically re-weight the contributions sourced from the U-Net's skip connections and backbone feature maps, to leverage the strengths of both components of the U-Net architecture. Promising results on image and video generation tasks demonstrate that our FreeU can be readily integrated to existing diffusion models, e.g., Stable Diffusion, DreamBooth, ModelScope, Rerender and ReVersion, to improve the generation quality with only a few lines of code. All you need is to adjust two scaling factors during inference. Project page: https://chenyangsi.top/FreeU/.

FreeU：在Diffusion U-Net中的免費午餐

FreeU: Free Lunch in Diffusion U-Net

摘要

Support