LU-NeRF：通过同步本地未定位的NeRF实现场景和姿势估计

LU-NeRF: Scene and Pose Estimation by Synchronizing Local Unposed NeRFs

June 8, 2023

作者: Zezhou Cheng, Carlos Esteves, Varun Jampani, Abhishek Kar, Subhransu Maji, Ameesh Makadia

cs.AI

摘要

阻碍 NeRF 模型广泛应用的一个关键障碍是它们对准确相机姿势的依赖。因此，人们越来越感兴趣将 NeRF 模型扩展到联合优化相机姿势和场景表示，这为通常使用已知失败模式的现成 SfM 管道提供了另一种选择。现有的未定位 NeRF 方法在有限的假设下运行，例如先验姿势分布或粗略姿势初始化，使它们在一般设置下效果较差。在这项工作中，我们提出了一种新颖的方法，LU-NeRF，它在对姿势配置放宽的情况下联合估计相机姿势和神经辐射场。我们的方法以局部到全局的方式运作，首先在数据的局部子集上进行优化，称为小场景。LU-NeRF 为这一具有挑战性的少样本任务估计局部姿势和几何。通过稳健的姿势同步步骤，将小场景姿势带入全局参考框架，最终可以执行姿势和场景的全局优化。我们展示了我们的 LU-NeRF 管道优于以往的未定位 NeRF 尝试，而无需对姿势先验进行限制性假设。这使我们能够在一般的 SE(3) 姿势设置下运行，而不像基线。我们的结果还表明，我们的模型可以作为基于特征的 SfM 管道的补充，因为它在低纹理和低分辨率图像上与 COLMAP 相比表现出色。

English

A critical obstacle preventing NeRF models from being deployed broadly in the wild is their reliance on accurate camera poses. Consequently, there is growing interest in extending NeRF models to jointly optimize camera poses and scene representation, which offers an alternative to off-the-shelf SfM pipelines which have well-understood failure modes. Existing approaches for unposed NeRF operate under limited assumptions, such as a prior pose distribution or coarse pose initialization, making them less effective in a general setting. In this work, we propose a novel approach, LU-NeRF, that jointly estimates camera poses and neural radiance fields with relaxed assumptions on pose configuration. Our approach operates in a local-to-global manner, where we first optimize over local subsets of the data, dubbed mini-scenes. LU-NeRF estimates local pose and geometry for this challenging few-shot task. The mini-scene poses are brought into a global reference frame through a robust pose synchronization step, where a final global optimization of pose and scene can be performed. We show our LU-NeRF pipeline outperforms prior attempts at unposed NeRF without making restrictive assumptions on the pose prior. This allows us to operate in the general SE(3) pose setting, unlike the baselines. Our results also indicate our model can be complementary to feature-based SfM pipelines as it compares favorably to COLMAP on low-texture and low-resolution images.