DATENeRF：NeRFs 的深度感知文字編輯

DATENeRF: Depth-Aware Text-based Editing of NeRFs

April 6, 2024

作者: Sara Rojas, Julien Philip, Kai Zhang, Sai Bi, Fujun Luan, Bernard Ghanem, Kalyan Sunkavall

cs.AI

摘要

最近擴散模型的進展在基於文本提示編輯2D圖像方面展現出卓越的能力。然而，將這些技術擴展到編輯神經輻射場（NeRF）中的場景是複雜的，因為編輯單獨的2D幀可能導致在多個視角之間出現不一致。我們的關鍵見解是，NeRF場景的幾何形狀可以作為整合這些2D編輯的橋樑。利用這種幾何形狀，我們使用一個深度條件的ControlNet來增強每個2D圖像修改的一致性。此外，我們引入了一種修補方法，利用NeRF場景的深度信息將2D編輯分佈到不同的圖像中，確保對錯誤和重採樣挑戰的魯棒性。我們的結果顯示，這種方法比現有的主導方法在基於文本的NeRF場景編輯方面實現了更一致、逼真和詳細的編輯。

English

Recent advancements in diffusion models have shown remarkable proficiency in editing 2D images based on text prompts. However, extending these techniques to edit scenes in Neural Radiance Fields (NeRF) is complex, as editing individual 2D frames can result in inconsistencies across multiple views. Our crucial insight is that a NeRF scene's geometry can serve as a bridge to integrate these 2D edits. Utilizing this geometry, we employ a depth-conditioned ControlNet to enhance the coherence of each 2D image modification. Moreover, we introduce an inpainting approach that leverages the depth information of NeRF scenes to distribute 2D edits across different images, ensuring robustness against errors and resampling challenges. Our results reveal that this methodology achieves more consistent, lifelike, and detailed edits than existing leading methods for text-driven NeRF scene editing.