RealFill：基於參考的真實圖像補全生成

RealFill: Reference-Driven Generation for Authentic Image Completion

September 28, 2023

作者: Luming Tang, Nataniel Ruiz, Qinghao Chu, Yuanzhen Li, Aleksander Holynski, David E. Jacobs, Bharath Hariharan, Yael Pritch, Neal Wadhwa, Kfir Aberman, Michael Rubinstein

cs.AI

摘要

最近在生成圖像方面的進展帶來了能夠在未知區域生成高質量、貌似真實的圖像內容的外部繪製和內部修補模型，但這些模型幻想的內容必然是不真實的，因為這些模型缺乏有關真實場景的足夠上下文。在這項工作中，我們提出了RealFill，一種新穎的生成方法，用於圖像完成，可以填補圖像中缺失的區域，並填充應該存在的內容。RealFill是一種生成修補模型，僅使用少量場景參考圖像進行個性化。這些參考圖像不必與目標圖像對齊，可以使用截然不同的視角、照明條件、相機光圈或圖像風格拍攝。一旦個性化，RealFill能夠以視覺上引人注目的內容完成目標圖像，並忠實於原始場景。我們在一個新的圖像完成基準測試集上評估了RealFill，該測試集涵蓋了一系列多樣且具有挑戰性的情境，發現其在性能上遠遠優於現有方法。更多結果請參見我們的專案頁面：https://realfill.github.io

English

Recent advances in generative imagery have brought forth outpainting and inpainting models that can produce high-quality, plausible image content in unknown regions, but the content these models hallucinate is necessarily inauthentic, since the models lack sufficient context about the true scene. In this work, we propose RealFill, a novel generative approach for image completion that fills in missing regions of an image with the content that should have been there. RealFill is a generative inpainting model that is personalized using only a few reference images of a scene. These reference images do not have to be aligned with the target image, and can be taken with drastically varying viewpoints, lighting conditions, camera apertures, or image styles. Once personalized, RealFill is able to complete a target image with visually compelling contents that are faithful to the original scene. We evaluate RealFill on a new image completion benchmark that covers a set of diverse and challenging scenarios, and find that it outperforms existing approaches by a large margin. See more results on our project page: https://realfill.github.io