改进相机姿势和分解低秩张量辐射场的联合优化的鲁棒性

Improving Robustness for Joint Optimization of Camera Poses and Decomposed Low-Rank Tensorial Radiance Fields

February 20, 2024

作者: Bo-Yu Cheng, Wei-Chen Chiu, Yu-Lun Liu

cs.AI

摘要

在本文中，我们提出了一种算法，允许联合优化由分解的低秩张量表示的相机姿态和场景几何，仅利用2D图像作为监督。首先，我们进行了基于1D信号的试点研究，并将我们的发现与3D场景联系起来，其中基于体素的NeRFs的天真联合姿态优化可能很容易导致次优解。此外，基于频谱分析，我们建议在2D和3D辐射场上应用卷积高斯滤波器，以实现粗到细的训练计划，从而实现联合相机姿态优化。利用分解的低秩张量中的分解属性，我们的方法实现了与蛮力3D卷积等效的效果，仅带来少量计算开销。为了进一步提高联合优化的鲁棒性和稳定性，我们还提出了平滑的2D监督技术、随机缩放的核参数以及边缘引导损失掩模的技术。广泛的定量和定性评估表明，我们提出的框架在新视角合成以及优化的快速收敛方面表现出优越性能。

English

In this paper, we propose an algorithm that allows joint refinement of camera pose and scene geometry represented by decomposed low-rank tensor, using only 2D images as supervision. First, we conduct a pilot study based on a 1D signal and relate our findings to 3D scenarios, where the naive joint pose optimization on voxel-based NeRFs can easily lead to sub-optimal solutions. Moreover, based on the analysis of the frequency spectrum, we propose to apply convolutional Gaussian filters on 2D and 3D radiance fields for a coarse-to-fine training schedule that enables joint camera pose optimization. Leveraging the decomposition property in decomposed low-rank tensor, our method achieves an equivalent effect to brute-force 3D convolution with only incurring little computational overhead. To further improve the robustness and stability of joint optimization, we also propose techniques of smoothed 2D supervision, randomly scaled kernel parameters, and edge-guided loss mask. Extensive quantitative and qualitative evaluations demonstrate that our proposed framework achieves superior performance in novel view synthesis as well as rapid convergence for optimization.