混合式神经辐射场：零样本物体生成和融合

Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields

June 22, 2023

作者: Ori Gordon, Omri Avrahami, Dani Lischinski

cs.AI

摘要

在由 NeRF 表示的 3D 场景中编辑本地区域或特定对象具有挑战性，主要是由于场景表示的隐式特性。将新的逼真对象一致地融入场景中增加了额外的难度。我们提出了 Blended-NeRF，这是一个强大而灵活的框架，用于根据文本提示或图像补丁以及 3D ROI 区域编辑现有 NeRF 场景中感兴趣的特定区域。我们的方法利用预训练的语言-图像模型来引导合成朝向用户提供的文本提示或图像补丁，同时结合一个在现有 NeRF 场景上初始化的 3D MLP 模型来生成对象并将其融入原始场景中的指定区域。我们通过在输入场景中定位 3D ROI 区域来实现本地编辑，并利用一种新颖的体积混合技术将 ROI 内部合成的内容与现有场景无缝融合。为了获得自然且视角一致的结果，我们利用现有和新的几何先验知识以及 3D 增强技术来提高最终结果的视觉保真度。 我们在各种真实 3D 场景和文本提示上定性和定量地测试了我们的框架，展示了与基线相比具有更大灵活性和多样性的逼真多视角一致结果。最后，我们展示了我们的框架在多个 3D 编辑应用中的适用性，包括向场景添加新对象、删除/替换/修改现有对象和纹理转换。

English

Editing a local region or a specific object in a 3D scene represented by a NeRF is challenging, mainly due to the implicit nature of the scene representation. Consistently blending a new realistic object into the scene adds an additional level of difficulty. We present Blended-NeRF, a robust and flexible framework for editing a specific region of interest in an existing NeRF scene, based on text prompts or image patches, along with a 3D ROI box. Our method leverages a pretrained language-image model to steer the synthesis towards a user-provided text prompt or image patch, along with a 3D MLP model initialized on an existing NeRF scene to generate the object and blend it into a specified region in the original scene. We allow local editing by localizing a 3D ROI box in the input scene, and seamlessly blend the content synthesized inside the ROI with the existing scene using a novel volumetric blending technique. To obtain natural looking and view-consistent results, we leverage existing and new geometric priors and 3D augmentations for improving the visual fidelity of the final result. We test our framework both qualitatively and quantitatively on a variety of real 3D scenes and text prompts, demonstrating realistic multi-view consistent results with much flexibility and diversity compared to the baselines. Finally, we show the applicability of our framework for several 3D editing applications, including adding new objects to a scene, removing/replacing/altering existing objects, and texture conversion.