NeRFiller：通過生成式3D修補完成場景

NeRFiller: Completing Scenes via Generative 3D Inpainting

December 7, 2023

作者: Ethan Weber, Aleksander Hołyński, Varun Jampani, Saurabh Saxena, Noah Snavely, Abhishek Kar, Angjoo Kanazawa

cs.AI

摘要

我們提出了 NeRFiller，一種通過使用現成的 2D 視覺生成模型進行生成式 3D 填充來完成 3D 捕獲中缺失部分的方法。通常，由於網格重建失敗或觀察不足（例如接觸區域，如物體底部或難以觸及的區域），捕獲的 3D 場景或物體的某些部分會缺失。我們通過利用 2D 填充擴散模型來應對這個具有挑戰性的 3D 填充問題。我們識別了這些模型的一個令人驚訝的行為，即當圖像形成 2x2 網格時，它們生成更具 3D 一致性的填充，並展示了如何將此行為推廣到超過四個圖像。然後，我們提出了一個迭代框架，將這些填充區域提煉成一個一致的 3D 場景。與相關作品相比，我們專注於完成場景而不是刪除前景物體，我們的方法不需要緊密的 2D 物體遮罩或文本。我們在各種場景上將我們的方法與適應我們設置的相關基準進行比較，其中 NeRFiller 創建了最具 3D 一致性和可信度的場景完成。我們的項目頁面位於 https://ethanweber.me/nerfiller。

English

We propose NeRFiller, an approach that completes missing portions of a 3D capture via generative 3D inpainting using off-the-shelf 2D visual generative models. Often parts of a captured 3D scene or object are missing due to mesh reconstruction failures or a lack of observations (e.g., contact regions, such as the bottom of objects, or hard-to-reach areas). We approach this challenging 3D inpainting problem by leveraging a 2D inpainting diffusion model. We identify a surprising behavior of these models, where they generate more 3D consistent inpaints when images form a 2times2 grid, and show how to generalize this behavior to more than four images. We then present an iterative framework to distill these inpainted regions into a single consistent 3D scene. In contrast to related works, we focus on completing scenes rather than deleting foreground objects, and our approach does not require tight 2D object masks or text. We compare our approach to relevant baselines adapted to our setting on a variety of scenes, where NeRFiller creates the most 3D consistent and plausible scene completions. Our project page is at https://ethanweber.me/nerfiller.