提升相機姿勢和分解低秩張量輻射場的聯合優化的魯棒性

Improving Robustness for Joint Optimization of Camera Poses and Decomposed Low-Rank Tensorial Radiance Fields

February 20, 2024

作者: Bo-Yu Cheng, Wei-Chen Chiu, Yu-Lun Liu

cs.AI

摘要

本文提出了一種演算法，允許使用僅2D影像監督的方式聯合優化由分解的低秩張量表示的相機姿態和場景幾何。首先，我們基於1D信號進行了一項試驗性研究，並將我們的發現與3D情境相關聯，其中對基於體素的 NeRFs 進行天真的聯合姿態優化可能會導致次優解。此外，基於頻譜分析，我們建議對2D和3D輻射場應用卷積高斯濾波器，以實現從粗糙到精細的訓練時間表，從而實現聯合相機姿態優化。利用分解低秩張量中的分解特性，我們的方法實現了與暴力3D卷積等效的效果，僅帶來少量計算開銷。為了進一步提高聯合優化的魯棒性和穩定性，我們還提出了平滑的2D監督技術、隨機縮放的核參數以及邊緣引導損失遮罩的技巧。廣泛的定量和定性評估表明，我們提出的框架在新視角合成和優化的快速收斂方面實現了卓越的性能。

English

In this paper, we propose an algorithm that allows joint refinement of camera pose and scene geometry represented by decomposed low-rank tensor, using only 2D images as supervision. First, we conduct a pilot study based on a 1D signal and relate our findings to 3D scenarios, where the naive joint pose optimization on voxel-based NeRFs can easily lead to sub-optimal solutions. Moreover, based on the analysis of the frequency spectrum, we propose to apply convolutional Gaussian filters on 2D and 3D radiance fields for a coarse-to-fine training schedule that enables joint camera pose optimization. Leveraging the decomposition property in decomposed low-rank tensor, our method achieves an equivalent effect to brute-force 3D convolution with only incurring little computational overhead. To further improve the robustness and stability of joint optimization, we also propose techniques of smoothed 2D supervision, randomly scaled kernel parameters, and edge-guided loss mask. Extensive quantitative and qualitative evaluations demonstrate that our proposed framework achieves superior performance in novel view synthesis as well as rapid convergence for optimization.