MonoPatchNeRF：利用基於 Patch 的單眼引導改進神經輻射場

MonoPatchNeRF: Improving Neural Radiance Fields with Patch-based Monocular Guidance

April 12, 2024

作者: Yuqun Wu, Jae Yong Lee, Chuhang Zou, Shenlong Wang, Derek Hoiem

cs.AI

摘要

最新的正則化神經輝度場（NeRF）方法在多視角立體（MVS）基準測試如ETH3D中產生了較差的幾何和視角外推。本文旨在創建提供準確幾何和視角合成的3D模型，部分彌合了NeRF與傳統MVS方法之間的巨大幾何性能差距。我們提出了一種基於補丁的方法，有效地利用單眼表面法向量和相對深度預測。基於補丁的射線採樣還實現了正規化互相關（NCC）和結構相似性（SSIM）之間隨機採樣的虛擬和訓練視圖的外觀正則化。我們進一步展示了基於稀疏結構從運動點的“密度限制”可以在稍微降低新視角合成指標的情況下，大大提高幾何準確性。我們的實驗顯示，在ETH3D MVS基準測試的平均F1@2cm上，我們的性能是RegNeRF的4倍，FreeNeRF的8倍，這表明了改善基於NeRF模型的幾何準確性是一個富有成果的研究方向，並為實現NeRF優於傳統MVS的最終優化方法提供了啟示。

English

The latest regularized Neural Radiance Field (NeRF) approaches produce poor geometry and view extrapolation for multiview stereo (MVS) benchmarks such as ETH3D. In this paper, we aim to create 3D models that provide accurate geometry and view synthesis, partially closing the large geometric performance gap between NeRF and traditional MVS methods. We propose a patch-based approach that effectively leverages monocular surface normal and relative depth predictions. The patch-based ray sampling also enables the appearance regularization of normalized cross-correlation (NCC) and structural similarity (SSIM) between randomly sampled virtual and training views. We further show that "density restrictions" based on sparse structure-from-motion points can help greatly improve geometric accuracy with a slight drop in novel view synthesis metrics. Our experiments show 4x the performance of RegNeRF and 8x that of FreeNeRF on average F1@2cm for ETH3D MVS benchmark, suggesting a fruitful research direction to improve the geometric accuracy of NeRF-based models, and sheds light on a potential future approach to enable NeRF-based optimization to eventually outperform traditional MVS.