MonoPatchNeRF：利用基于补丁的单目引导改进神经辐射场

MonoPatchNeRF: Improving Neural Radiance Fields with Patch-based Monocular Guidance

April 12, 2024

作者: Yuqun Wu, Jae Yong Lee, Chuhang Zou, Shenlong Wang, Derek Hoiem

cs.AI

摘要

最新的正则化神经辐射场（NeRF）方法在多视图立体（MVS）基准测试中（如ETH3D）产生了几何和视角外推方面的问题。本文旨在创建提供准确几何和视角合成的3D模型，部分弥合NeRF和传统MVS方法之间的巨大几何性能差距。我们提出了一种基于补丁的方法，有效地利用单眼表面法线和相对深度预测。基于补丁的射线采样还能够在随机采样的虚拟视图和训练视图之间实现外观正则化，采用了归一化互相关（NCC）和结构相似性（SSIM）。我们进一步表明，基于稀疏的结构运动点的“密度限制”可以在轻微降低新颖视图合成指标的情况下极大地提高几何精度。我们的实验结果显示，在ETH3D MVS基准测试的平均F1@2cm上，性能比RegNeRF高出4倍，比FreeNeRF高出8倍，这表明了一个富有成果的研究方向，可以改善基于NeRF的模型的几何精度，并为实现NeRF优于传统MVS方法提供了潜在的未来途径。

English

The latest regularized Neural Radiance Field (NeRF) approaches produce poor geometry and view extrapolation for multiview stereo (MVS) benchmarks such as ETH3D. In this paper, we aim to create 3D models that provide accurate geometry and view synthesis, partially closing the large geometric performance gap between NeRF and traditional MVS methods. We propose a patch-based approach that effectively leverages monocular surface normal and relative depth predictions. The patch-based ray sampling also enables the appearance regularization of normalized cross-correlation (NCC) and structural similarity (SSIM) between randomly sampled virtual and training views. We further show that "density restrictions" based on sparse structure-from-motion points can help greatly improve geometric accuracy with a slight drop in novel view synthesis metrics. Our experiments show 4x the performance of RegNeRF and 8x that of FreeNeRF on average F1@2cm for ETH3D MVS benchmark, suggesting a fruitful research direction to improve the geometric accuracy of NeRF-based models, and sheds light on a potential future approach to enable NeRF-based optimization to eventually outperform traditional MVS.