引导与重缩放：实现高效免调优真实图像编辑的自引导机制

Guide-and-Rescale: Self-Guidance Mechanism for Effective Tuning-Free Real Image Editing

September 2, 2024

作者: Vadim Titov, Madina Khalmatova, Alexandra Ivanova, Dmitry Vetrov, Aibek Alanov

cs.AI

摘要

尽管大规模文本到图像生成模型近期取得了显著进展，利用这些模型对真实图像进行操控仍是一个难题。现有编辑方法的主要局限在于，它们要么无法在广泛的图像编辑任务中保持一致的品质，要么需要耗费大量时间进行超参数调优或扩散模型的微调，以保留输入图像特有的外观。我们提出了一种新颖的方法，该方法基于通过引导机制改进的扩散采样过程。在本研究中，我们探索了自引导技术，旨在保留输入图像的整体结构及其不应被编辑的局部区域外观。具体而言，我们明确引入了旨在保存源图像局部与全局结构的布局保持能量函数。此外，我们提出了一种噪声重缩放机制，通过在生成过程中平衡无分类器引导与我们提出的引导器的范数，来保持噪声分布。这种引导方法无需对扩散模型进行微调，也无需精确的反转过程。因此，所提出的方法提供了一种快速且高质量的编辑机制。在我们的实验中，通过人类评估与定量分析，我们展示了该方法能够生成更受人类青睐的期望编辑效果，并在编辑质量与原始图像保留之间实现了更好的平衡。我们的代码可在https://github.com/FusionBrainLab/Guide-and-Rescale获取。

English

Despite recent advances in large-scale text-to-image generative models, manipulating real images with these models remains a challenging problem. The main limitations of existing editing methods are that they either fail to perform with consistent quality on a wide range of image edits or require time-consuming hyperparameter tuning or fine-tuning of the diffusion model to preserve the image-specific appearance of the input image. We propose a novel approach that is built upon a modified diffusion sampling process via the guidance mechanism. In this work, we explore the self-guidance technique to preserve the overall structure of the input image and its local regions appearance that should not be edited. In particular, we explicitly introduce layout-preserving energy functions that are aimed to save local and global structures of the source image. Additionally, we propose a noise rescaling mechanism that allows to preserve noise distribution by balancing the norms of classifier-free guidance and our proposed guiders during generation. Such a guiding approach does not require fine-tuning the diffusion model and exact inversion process. As a result, the proposed method provides a fast and high-quality editing mechanism. In our experiments, we show through human evaluation and quantitative analysis that the proposed method allows to produce desired editing which is more preferable by humans and also achieves a better trade-off between editing quality and preservation of the original image. Our code is available at https://github.com/FusionBrainLab/Guide-and-Rescale.