HD-Painter：高分辨率且快速可信的文本引導圖像修補與擴散模型
HD-Painter: High-Resolution and Prompt-Faithful Text-Guided Image Inpainting with Diffusion Models
December 21, 2023
作者: Hayk Manukyan, Andranik Sargsyan, Barsegh Atanyan, Zhangyang Wang, Shant Navasardyan, Humphrey Shi
cs.AI
摘要
最近在基於文本引導的圖像修補方面取得了顯著進展，這是基於文本到圖像擴散模型取得了前所未有的成功，產生了極為逼真和視覺上可信的結果。然而，目前的文本到圖像修補模型仍有顯著的改進潛力，特別是在更好地對齊修補區域與用戶提示以及進行高分辨率修補方面。因此，在本文中，我們介紹了HD-Painter，這是一種完全無需訓練的方法，可以準確地遵循提示並且能夠一致地擴展到高分辨率圖像修補。為此，我們設計了Prompt-Aware Introverted Attention（PAIntA）層，通過提示信息增強自注意力分數，從而產生更好的文本對齊生成。為了進一步提高提示的一致性，我們引入了Reweighting Attention Score Guidance（RASG）機制，無縫地將一種事後抽樣策略整合到DDIM的一般形式中，以防止分布外潛在變化。此外，HD-Painter通過引入一種專門為修補定制的超分辨率技術，使其能夠擴展到更大的比例，實現對高達2K分辨率的圖像中缺失區域的完成。我們的實驗表明，HD-Painter在質量和量化上均優於現有的最先進方法，實現了61.4%對51.9%的令人印象深刻的生成準確度改進。我們將在以下網址公開提供代碼：https://github.com/Picsart-AI-Research/HD-Painter
English
Recent progress in text-guided image inpainting, based on the unprecedented
success of text-to-image diffusion models, has led to exceptionally realistic
and visually plausible results. However, there is still significant potential
for improvement in current text-to-image inpainting models, particularly in
better aligning the inpainted area with user prompts and performing
high-resolution inpainting. Therefore, in this paper we introduce HD-Painter, a
completely training-free approach that accurately follows to prompts and
coherently scales to high-resolution image inpainting. To this end, we design
the Prompt-Aware Introverted Attention (PAIntA) layer enhancing self-attention
scores by prompt information and resulting in better text alignment
generations. To further improve the prompt coherence we introduce the
Reweighting Attention Score Guidance (RASG) mechanism seamlessly integrating a
post-hoc sampling strategy into general form of DDIM to prevent
out-of-distribution latent shifts. Moreover, HD-Painter allows extension to
larger scales by introducing a specialized super-resolution technique
customized for inpainting, enabling the completion of missing regions in images
of up to 2K resolution. Our experiments demonstrate that HD-Painter surpasses
existing state-of-the-art approaches qualitatively and quantitatively,
achieving an impressive generation accuracy improvement of 61.4% vs 51.9%. We
will make the codes publicly available at:
https://github.com/Picsart-AI-Research/HD-Painter