DreamEditor：使用神經場實現基於文本的3D場景編輯

DreamEditor: Text-Driven 3D Scene Editing with Neural Fields

June 23, 2023

作者: Jingyu Zhuang, Chen Wang, Lingjie Liu, Liang Lin, Guanbin Li

cs.AI

摘要

神經場在視圖合成和場景重建方面取得了令人印象深刻的進展。然而，由於幾何和紋理信息的隱式編碼，編輯這些神經場仍然具有挑戰性。在本文中，我們提出了一個新穎的框架DreamEditor，使用者可以通過文本提示對神經場進行受控編輯。通過將場景表示為基於網格的神經場，DreamEditor允許在特定區域內進行局部編輯。DreamEditor利用預訓練的文本到圖像擴散模型的文本編碼器，根據文本提示的語義自動識別要編輯的區域。隨後，DreamEditor通過分數蒸餾取樣對編輯區域進行優化，並將其幾何和紋理與文本提示對齊。大量實驗表明，DreamEditor能夠根據給定的文本提示準確編輯現實場景的神經場，同時確保無關區域的一致性。DreamEditor生成高度逼真的紋理和幾何，明顯超越先前的作品在定量和定性評估方面。

English

Neural fields have achieved impressive advancements in view synthesis and scene reconstruction. However, editing these neural fields remains challenging due to the implicit encoding of geometry and texture information. In this paper, we propose DreamEditor, a novel framework that enables users to perform controlled editing of neural fields using text prompts. By representing scenes as mesh-based neural fields, DreamEditor allows localized editing within specific regions. DreamEditor utilizes the text encoder of a pretrained text-to-Image diffusion model to automatically identify the regions to be edited based on the semantics of the text prompts. Subsequently, DreamEditor optimizes the editing region and aligns its geometry and texture with the text prompts through score distillation sampling [29]. Extensive experiments have demonstrated that DreamEditor can accurately edit neural fields of real-world scenes according to the given text prompts while ensuring consistency in irrelevant areas. DreamEditor generates highly realistic textures and geometry, significantly surpassing previous works in both quantitative and qualitative evaluations.