ICON：用於聯合姿勢和輝度場增量信心的優化

ICON: Incremental CONfidence for Joint Pose and Radiance Field Optimization

January 17, 2024

作者: Weiyao Wang, Pierre Gleize, Hao Tang, Xingyu Chen, Kevin J Liang, Matt Feiszli

cs.AI

摘要

神經輻射場（Neural Radiance Fields，NeRF）在給定一組2D圖像時，對於新視角合成（Novel View Synthesis，NVS）表現出卓越的性能。然而，NeRF的訓練需要對每個輸入視角進行準確的相機姿勢，通常通過從運動中恢復（Structure-from-Motion，SfM）管道獲取。最近的研究嘗試放寬這種限制，但它們仍然常常依賴可以進行微調的良好初始姿勢。在這裡，我們的目標是消除對姿勢初始化的要求。我們提出了增量信心（Incremental CONfidence，ICON），這是一種從2D視頻幀訓練NeRF的優化程序。ICON僅假設平滑的相機運動來估計姿勢的初始猜測。此外，ICON引入了“信心”：一種用於動態重新加權梯度的模型質量自適應度量。ICON依賴於高信心的姿勢來學習NeRF，並依賴於高信心的3D結構（由NeRF編碼）來學習姿勢。我們展示了，ICON在沒有先前姿勢初始化的情況下，在CO3D和HO3D方面的性能優於使用SfM姿勢的方法。

English

Neural Radiance Fields (NeRF) exhibit remarkable performance for Novel View Synthesis (NVS) given a set of 2D images. However, NeRF training requires accurate camera pose for each input view, typically obtained by Structure-from-Motion (SfM) pipelines. Recent works have attempted to relax this constraint, but they still often rely on decent initial poses which they can refine. Here we aim at removing the requirement for pose initialization. We present Incremental CONfidence (ICON), an optimization procedure for training NeRFs from 2D video frames. ICON only assumes smooth camera motion to estimate initial guess for poses. Further, ICON introduces ``confidence": an adaptive measure of model quality used to dynamically reweight gradients. ICON relies on high-confidence poses to learn NeRF, and high-confidence 3D structure (as encoded by NeRF) to learn poses. We show that ICON, without prior pose initialization, achieves superior performance in both CO3D and HO3D versus methods which use SfM pose.