SplatFields：用于稀疏3D和4D重建的神经高斯斑点

SplatFields: Neural Gaussian Splats for Sparse 3D and 4D Reconstruction

September 17, 2024

作者: Marko Mihajlovic, Sergey Prokudin, Siyu Tang, Robert Maier, Federica Bogo, Tony Tung, Edmond Boyer

cs.AI

摘要

在计算机视觉和图形学中，从多视图图像中数字化3D静态场景和4D动态事件长期以来一直是一个挑战。最近，3D高斯飞溅（3DGS）已经成为一种实用且可扩展的重建方法，因其令人印象深刻的重建质量、实时渲染能力以及与广泛使用的可视化工具兼容而备受青睐。然而，该方法需要大量的输入视图才能实现高质量的场景重建，这引入了一个重要的实际瓶颈。在捕捉动态场景时，部署大规模摄像机阵列可能成本过高，这一挑战尤为严峻。在这项工作中，我们确定了高斯飞溅特征缺乏空间自相关性是导致3DGS技术在稀疏重建环境中表现不佳的因素之一。为解决这一问题，我们提出了一种优化策略，通过将其建模为相应的隐式神经场的输出，有效地规范化飞溅特征。这导致在各种场景中重建质量的一致提升。我们的方法有效处理静态和动态情况，通过在不同设置和场景复杂性下进行广泛测试加以证明。

English

Digitizing 3D static scenes and 4D dynamic events from multi-view images has long been a challenge in computer vision and graphics. Recently, 3D Gaussian Splatting (3DGS) has emerged as a practical and scalable reconstruction method, gaining popularity due to its impressive reconstruction quality, real-time rendering capabilities, and compatibility with widely used visualization tools. However, the method requires a substantial number of input views to achieve high-quality scene reconstruction, introducing a significant practical bottleneck. This challenge is especially severe in capturing dynamic scenes, where deploying an extensive camera array can be prohibitively costly. In this work, we identify the lack of spatial autocorrelation of splat features as one of the factors contributing to the suboptimal performance of the 3DGS technique in sparse reconstruction settings. To address the issue, we propose an optimization strategy that effectively regularizes splat features by modeling them as the outputs of a corresponding implicit neural field. This results in a consistent enhancement of reconstruction quality across various scenarios. Our approach effectively handles static and dynamic cases, as demonstrated by extensive testing across different setups and scene complexities.