InseRF：神經3D場景中基於文本驅動的生成式物體插入

摘要

我們介紹了一種新方法InseRF，用於在3D場景的NeRF重建中生成物件插入。基於用戶提供的文本描述和參考視角中的2D邊界框，InseRF在3D場景中生成新物件。最近，由於在3D生成建模中使用了文本到圖像擴散模型的強先驗，對於3D場景編輯的方法已經發生了深刻的變革。現有方法主要有效地通過風格和外觀變化或刪除現有物件來編輯3D場景。然而，對於這些方法來說，生成新物件仍然是一個挑戰，我們在本研究中解決了這個問題。具體來說，我們建議將3D物件插入基於參考視角的2D物件插入。然後，通過單視圖物件重建方法將2D編輯提升到3D。然後，在導向單眼深度估計方法的先驗指導下，將重建的物件插入到場景中。我們在各種3D場景上評估了我們的方法，並對所提出的組件進行了深入分析。我們在幾個3D場景中進行的生成物件插入實驗表明，與現有方法相比，InseRF的效果顯著。InseRF能夠實現可控且3D一致的物件插入，而無需作為輸入的明確3D信息。請訪問我們的項目頁面：https://mohamad-shahbazi.github.io/inserf。

English

We introduce InseRF, a novel method for generative object insertion in the NeRF reconstructions of 3D scenes. Based on a user-provided textual description and a 2D bounding box in a reference viewpoint, InseRF generates new objects in 3D scenes. Recently, methods for 3D scene editing have been profoundly transformed, owing to the use of strong priors of text-to-image diffusion models in 3D generative modeling. Existing methods are mostly effective in editing 3D scenes via style and appearance changes or removing existing objects. Generating new objects, however, remains a challenge for such methods, which we address in this study. Specifically, we propose grounding the 3D object insertion to a 2D object insertion in a reference view of the scene. The 2D edit is then lifted to 3D using a single-view object reconstruction method. The reconstructed object is then inserted into the scene, guided by the priors of monocular depth estimation methods. We evaluate our method on various 3D scenes and provide an in-depth analysis of the proposed components. Our experiments with generative insertion of objects in several 3D scenes indicate the effectiveness of our method compared to the existing methods. InseRF is capable of controllable and 3D-consistent object insertion without requiring explicit 3D information as input. Please visit our project page at https://mohamad-shahbazi.github.io/inserf.

InseRF：神經3D場景中基於文本驅動的生成式物體插入

InseRF: Text-Driven Generative Object Insertion in Neural 3D Scenes

摘要

Support