HD-Painter：高分辨率和快速准确的文本引导图像修复与扩散模型
HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models
December 21, 2023
作者: Hayk Manukyan, Andranik Sargsyan, Barsegh Atanyan, Zhangyang Wang, Shant Navasardyan, Humphrey Shi
cs.AI
摘要
最近在文本引导的图像修复方面取得的进展，基于文本到图像扩散模型的空前成功，导致了异常逼真和视觉上可信的结果。然而，目前的文本到图像修复模型仍有显著的改进潜力，特别是在更好地将修复区域与用户提示对齐以及进行高分辨率修复方面。因此，在本文中，我们介绍了HD-Painter，这是一种完全无需训练的方法，能够准确地遵循提示并一致地扩展到高分辨率图像修复。为此，我们设计了Prompt-Aware Introverted Attention（PAIntA）层，通过提示信息增强自注意力分数，从而产生更好的文本对齐生成结果。为了进一步提高提示的连贯性，我们引入了Reweighting Attention Score Guidance（RASG）机制，将一种事后采样策略无缝集成到DDIM的一般形式中，以防止分布外的潜在偏移。此外，HD-Painter通过引入一种针对修复的专门超分辨技术，允许扩展到更大的尺度，能够完成高达2K分辨率的图像中缺失区域的修复。我们的实验表明，HD-Painter在质量和数量上均超越了现有的最先进方法，实现了惊人的生成准确度提高，为61.4% vs 51.9%。我们将在以下网址公开提供代码：https://github.com/Picsart-AI-Research/HD-Painter
English
Recent progress in text-guided image inpainting, based on the unprecedented
success of text-to-image diffusion models, has led to exceptionally realistic
and visually plausible results. However, there is still significant potential
for improvement in current text-to-image inpainting models, particularly in
better aligning the inpainted area with user prompts and performing
high-resolution inpainting. Therefore, in this paper we introduce HD-Painter, a
completely training-free approach that accurately follows to prompts and
coherently scales to high-resolution image inpainting. To this end, we design
the Prompt-Aware Introverted Attention (PAIntA) layer enhancing self-attention
scores by prompt information and resulting in better text alignment
generations. To further improve the prompt coherence we introduce the
Reweighting Attention Score Guidance (RASG) mechanism seamlessly integrating a
post-hoc sampling strategy into general form of DDIM to prevent
out-of-distribution latent shifts. Moreover, HD-Painter allows extension to
larger scales by introducing a specialized super-resolution technique
customized for inpainting, enabling the completion of missing regions in images
of up to 2K resolution. Our experiments demonstrate that HD-Painter surpasses
existing state-of-the-art approaches qualitatively and quantitatively,
achieving an impressive generation accuracy improvement of 61.4% vs 51.9%. We
will make the codes publicly available at:
https://github.com/Picsart-AI-Research/HD-Painter