驯化潜在扩散模型用于神经辐射场修复

Taming Latent Diffusion Model for Neural Radiance Field Inpainting

April 15, 2024

作者: Chieh Hubert Lin, Changil Kim, Jia-Bin Huang, Qinbo Li, Chih-Yao Ma, Johannes Kopf, Ming-Hsuan Yang, Hung-Yu Tseng

cs.AI

摘要

神经辐射场（NeRF）是一种从多视图图像进行3D重建的表示。尽管最近的一些工作展示了在扩散先验条件下编辑重建的NeRF取得了初步成功，但它们仍然难以合成完全未覆盖区域的合理几何形状。一个主要原因是扩散模型产生的合成内容具有高度多样性，这阻碍了辐射场收敛到清晰和确定性几何形状。此外，将潜在扩散模型应用于真实数据通常会由于自动编码错误导致图像条件不一致的纹理偏移。这两个问题进一步加剧了使用像素距离损失。为解决这些问题，我们提出通过每个场景的定制来调节扩散模型的随机性，并通过掩码对抗训练来减轻纹理偏移。在分析过程中，我们还发现常用的像素损失和感知损失在NeRF修复任务中是有害的。通过严格的实验，我们的框架在各种真实场景上实现了最先进的NeRF修复结果。项目页面：https://hubert0527.github.io/MALD-NeRF

English

Neural Radiance Field (NeRF) is a representation for 3D reconstruction from multi-view images. Despite some recent work showing preliminary success in editing a reconstructed NeRF with diffusion prior, they remain struggling to synthesize reasonable geometry in completely uncovered regions. One major reason is the high diversity of synthetic contents from the diffusion model, which hinders the radiance field from converging to a crisp and deterministic geometry. Moreover, applying latent diffusion models on real data often yields a textural shift incoherent to the image condition due to auto-encoding errors. These two problems are further reinforced with the use of pixel-distance losses. To address these issues, we propose tempering the diffusion model's stochasticity with per-scene customization and mitigating the textural shift with masked adversarial training. During the analyses, we also found the commonly used pixel and perceptual losses are harmful in the NeRF inpainting task. Through rigorous experiments, our framework yields state-of-the-art NeRF inpainting results on various real-world scenes. Project page: https://hubert0527.github.io/MALD-NeRF