UniSDF：统一神经表示以实现具有反射的复杂场景的高保真度3D重建

UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections

December 20, 2023

作者: Fangjinhua Wang, Marie-Julie Rakotosaona, Michael Niemeyer, Richard Szeliski, Marc Pollefeys, Federico Tombari

cs.AI

摘要

神经网络3D场景表示已展现出从2D图像重建3D场景的巨大潜力。然而，重建复杂场景的真实世界捕获仍然是一个挑战。现有的通用3D重建方法通常难以表现精细的几何细节，并且未能充分模拟大规模场景的反射表面。专注于反射表面的技术可以通过更好的反射参数化来模拟复杂和详细的反射。然而，我们观察到这些方法在真实的无界场景中通常不够稳健，因为存在非反射和反射组件。在这项工作中，我们提出了UniSDF，一种通用的3D重建方法，可以重建具有反射的大型复杂场景。我们研究了基于视图和基于反射的颜色预测参数化技术，并发现在3D空间中显式地融合这些表示可以实现更准确的几何表面重建，特别是对于反射表面。我们进一步将这种表示与以粗到细方式训练的多分辨率网格骨干相结合，使得重建速度比先前方法更快。在对象级数据集DTU、Shiny Blender以及无界数据集Mip-NeRF 360和Ref-NeRF real上进行了大量实验，证明我们的方法能够稳健地重建具有精细细节和反射表面的复杂大型场景。请访问我们的项目页面https://fangjinhuawang.github.io/UniSDF。

English

Neural 3D scene representations have shown great potential for 3D reconstruction from 2D images. However, reconstructing real-world captures of complex scenes still remains a challenge. Existing generic 3D reconstruction methods often struggle to represent fine geometric details and do not adequately model reflective surfaces of large-scale scenes. Techniques that explicitly focus on reflective surfaces can model complex and detailed reflections by exploiting better reflection parameterizations. However, we observe that these methods are often not robust in real unbounded scenarios where non-reflective as well as reflective components are present. In this work, we propose UniSDF, a general purpose 3D reconstruction method that can reconstruct large complex scenes with reflections. We investigate both view-based as well as reflection-based color prediction parameterization techniques and find that explicitly blending these representations in 3D space enables reconstruction of surfaces that are more geometrically accurate, especially for reflective surfaces. We further combine this representation with a multi-resolution grid backbone that is trained in a coarse-to-fine manner, enabling faster reconstructions than prior methods. Extensive experiments on object-level datasets DTU, Shiny Blender as well as unbounded datasets Mip-NeRF 360 and Ref-NeRF real demonstrate that our method is able to robustly reconstruct complex large-scale scenes with fine details and reflective surfaces. Please see our project page at https://fangjinhuawang.github.io/UniSDF.