DATENeRF：NeRF的深度感知文本编辑

DATENeRF: Depth-Aware Text-based Editing of NeRFs

April 6, 2024

作者: Sara Rojas, Julien Philip, Kai Zhang, Sai Bi, Fujun Luan, Bernard Ghanem, Kalyan Sunkavall

cs.AI

摘要

最近扩散模型的进展展示了在基于文本提示的情况下编辑2D图像方面的显著熟练度。然而，将这些技术扩展到编辑神经辐射场（NeRF）中的场景是复杂的，因为编辑单个2D帧可能导致多个视图之间的不一致性。我们的关键见解是，NeRF场景的几何形状可以作为整合这些2D编辑的桥梁。利用这种几何形状，我们使用一个深度条件的ControlNet来增强每个2D图像修改的连贯性。此外，我们引入了一种修补方法，利用NeRF场景的深度信息将2D编辑分布到不同的图像中，确保对错误和重采样挑战具有鲁棒性。我们的结果显示，这种方法比现有的主流基于文本的NeRF场景编辑方法实现了更一致、栩栩如生和详细的编辑。

English

Recent advancements in diffusion models have shown remarkable proficiency in editing 2D images based on text prompts. However, extending these techniques to edit scenes in Neural Radiance Fields (NeRF) is complex, as editing individual 2D frames can result in inconsistencies across multiple views. Our crucial insight is that a NeRF scene's geometry can serve as a bridge to integrate these 2D edits. Utilizing this geometry, we employ a depth-conditioned ControlNet to enhance the coherence of each 2D image modification. Moreover, we introduce an inpainting approach that leverages the depth information of NeRF scenes to distribute 2D edits across different images, ensuring robustness against errors and resampling challenges. Our results reveal that this methodology achieves more consistent, lifelike, and detailed edits than existing leading methods for text-driven NeRF scene editing.