UniSDF：統一神經表示以實現具有反射的複雜場景的高保真度3D重建

UniSDF: Unifying Neural Representations for High-Fidelity 3D Reconstruction of Complex Scenes with Reflections

December 20, 2023

作者: Fangjinhua Wang, Marie-Julie Rakotosaona, Michael Niemeyer, Richard Szeliski, Marc Pollefeys, Federico Tombari

cs.AI

摘要

神經3D場景表示已展現出從2D影像重建3D的巨大潛力。然而，復雜場景的真實世界捕獲重建仍然是一個挑戰。現有的通用3D重建方法通常難以表現精細的幾何細節，並且未能充分建模大型場景的反射表面。專注於反射表面的技術可以通過更好的反射參數化來建模複雜和詳細的反射。然而，我們觀察到這些方法在現實中存在非反射和反射組成的無界場景時通常不夠穩健。在這項工作中，我們提出了UniSDF，一種通用的3D重建方法，可以重建具有反射的大型複雜場景。我們研究了基於視圖和基於反射的顏色預測參數化技術，並發現在3D空間中明確地混合這些表示可以實現更加幾何精確的表面重建，尤其是對於反射表面。我們進一步將這種表示與以粗到細的方式訓練的多分辨率網格骨幹結合，實現比先前方法更快的重建速度。對DTU、Shiny Blender等對象級數據集以及無界數據集Mip-NeRF 360和Ref-NeRF real進行了大量實驗，證明我們的方法能夠穩健地重建具有精細細節和反射表面的複雜大型場景。請參閱我們的項目頁面：https://fangjinhuawang.github.io/UniSDF。

English

Neural 3D scene representations have shown great potential for 3D reconstruction from 2D images. However, reconstructing real-world captures of complex scenes still remains a challenge. Existing generic 3D reconstruction methods often struggle to represent fine geometric details and do not adequately model reflective surfaces of large-scale scenes. Techniques that explicitly focus on reflective surfaces can model complex and detailed reflections by exploiting better reflection parameterizations. However, we observe that these methods are often not robust in real unbounded scenarios where non-reflective as well as reflective components are present. In this work, we propose UniSDF, a general purpose 3D reconstruction method that can reconstruct large complex scenes with reflections. We investigate both view-based as well as reflection-based color prediction parameterization techniques and find that explicitly blending these representations in 3D space enables reconstruction of surfaces that are more geometrically accurate, especially for reflective surfaces. We further combine this representation with a multi-resolution grid backbone that is trained in a coarse-to-fine manner, enabling faster reconstructions than prior methods. Extensive experiments on object-level datasets DTU, Shiny Blender as well as unbounded datasets Mip-NeRF 360 and Ref-NeRF real demonstrate that our method is able to robustly reconstruct complex large-scale scenes with fine details and reflective surfaces. Please see our project page at https://fangjinhuawang.github.io/UniSDF.