LU-NeRF：通過同步本地未定位的 NeRF 進行場景和姿勢估計

LU-NeRF: Scene and Pose Estimation by Synchronizing Local Unposed NeRFs

June 8, 2023

作者: Zezhou Cheng, Carlos Esteves, Varun Jampani, Abhishek Kar, Subhransu Maji, Ameesh Makadia

cs.AI

摘要

阻礙 NeRF 模型在廣泛應用中的一個關鍵障礙是其對準確相機姿勢的依賴。因此，人們越來越感興趣擴展 NeRF 模型以共同優化相機姿勢和場景表示，這提供了一種替代方案，可以避免已知的 SfM 管道的失敗模式。現有的未定位 NeRF 方法在有限的假設下運作，例如先前的姿勢分佈或粗略的姿勢初始化，使其在一般情況下效果較差。在這項工作中，我們提出了一種新穎的方法 LU-NeRF，它可以聯合估計相機姿勢和神經輻射場，並對姿勢配置放鬆了假設。我們的方法以從局部到全局的方式運作，首先在數據的局部子集上進行優化，被稱為小場景。LU-NeRF 估計這個具有挑戰性的少樣本任務的局部姿勢和幾何。通過強健的姿勢同步步驟，將小場景姿勢帶入全局參考框架，最終可以執行姿勢和場景的全局優化。我們展示了我們的 LU-NeRF 管道優於先前的未定位 NeRF 嘗試，而無需對姿勢先前進行限制性假設。這使我們能夠在一般的 SE(3) 姿勢設置中運作，與基準線不同。我們的結果還表明，我們的模型可以與基於特徵的 SfM 管道互補，因為它在低紋理和低分辨率圖像上與 COLMAP 相比表現優越。

English

A critical obstacle preventing NeRF models from being deployed broadly in the wild is their reliance on accurate camera poses. Consequently, there is growing interest in extending NeRF models to jointly optimize camera poses and scene representation, which offers an alternative to off-the-shelf SfM pipelines which have well-understood failure modes. Existing approaches for unposed NeRF operate under limited assumptions, such as a prior pose distribution or coarse pose initialization, making them less effective in a general setting. In this work, we propose a novel approach, LU-NeRF, that jointly estimates camera poses and neural radiance fields with relaxed assumptions on pose configuration. Our approach operates in a local-to-global manner, where we first optimize over local subsets of the data, dubbed mini-scenes. LU-NeRF estimates local pose and geometry for this challenging few-shot task. The mini-scene poses are brought into a global reference frame through a robust pose synchronization step, where a final global optimization of pose and scene can be performed. We show our LU-NeRF pipeline outperforms prior attempts at unposed NeRF without making restrictive assumptions on the pose prior. This allows us to operate in the general SE(3) pose setting, unlike the baselines. Our results also indicate our model can be complementary to feature-based SfM pipelines as it compares favorably to COLMAP on low-texture and low-resolution images.