Splatfacto-W：高斯点喷洒在无约束照片集中的Nerfstudio实现

Splatfacto-W: A Nerfstudio Implementation of Gaussian Splatting for Unconstrained Photo Collections

July 17, 2024

作者: Congrong Xu, Justin Kerr, Angjoo Kanazawa

cs.AI

摘要

从无约束的野外图像集合中合成新视角仍然是一个重要且具有挑战性的任务，这是由于光度变化和瞬时遮挡物使准确的场景重建变得复杂。先前的方法通过在神经辐射场（NeRFs）中集成每个图像外观特征嵌入来解决这些问题。虽然3D高斯飞溅（3DGS）提供了更快的训练和实时渲染，但要将其适应无约束的图像集合并非易事，因为其架构存在显著不同。在本文中，我们介绍了Splatfacto-W，一种方法，它将每个高斯神经颜色特征和每个图像外观嵌入集成到光栅化过程中，同时采用基于球谐函数的背景模型来表示不同的光度外观并更好地描绘背景。我们的关键贡献包括潜在外观建模、高效的瞬时对象处理和精确的背景建模。Splatfacto-W在野外场景中提供了高质量、实时的新视角合成，改善了场景一致性。我们的方法将峰值信噪比（PSNR）平均提高了5.3 dB，比3DGS提高了150倍的训练速度，同时实现了与3DGS相似的渲染速度。更多视频结果和集成到Nerfstudio的代码可在https://kevinxu02.github.io/splatfactow/上找到。

English

Novel view synthesis from unconstrained in-the-wild image collections remains a significant yet challenging task due to photometric variations and transient occluders that complicate accurate scene reconstruction. Previous methods have approached these issues by integrating per-image appearance features embeddings in Neural Radiance Fields (NeRFs). Although 3D Gaussian Splatting (3DGS) offers faster training and real-time rendering, adapting it for unconstrained image collections is non-trivial due to the substantially different architecture. In this paper, we introduce Splatfacto-W, an approach that integrates per-Gaussian neural color features and per-image appearance embeddings into the rasterization process, along with a spherical harmonics-based background model to represent varying photometric appearances and better depict backgrounds. Our key contributions include latent appearance modeling, efficient transient object handling, and precise background modeling. Splatfacto-W delivers high-quality, real-time novel view synthesis with improved scene consistency in in-the-wild scenarios. Our method improves the Peak Signal-to-Noise Ratio (PSNR) by an average of 5.3 dB compared to 3DGS, enhances training speed by 150 times compared to NeRF-based methods, and achieves a similar rendering speed to 3DGS. Additional video results and code integrated into Nerfstudio are available at https://kevinxu02.github.io/splatfactow/.