MVGS：多视图调节高斯点云喷洒用于新视角合成

MVGS: Multi-view-regulated Gaussian Splatting for Novel View Synthesis

October 2, 2024

作者: Xiaobiao Du, Yida Wang, Xin Yu

cs.AI

摘要

最近关于体积渲染的研究，例如NeRF和3D高斯光斑（3DGS），通过学习的隐式神经辐射场或3D高斯函数显著提高了渲染质量和效率。在显式表示的基础上进行渲染，基本的3D高斯光斑及其变体通过在训练过程中每次迭代优化参数模型以实现实时效率，这一方法源自NeRF。因此，某些视角被过度拟合，导致新视角合成的外观不尽如人意，以及3D几何形状不够精确。为了解决上述问题，我们提出了一种新的3D高斯光斑优化方法，包含四个关键的创新贡献：1）我们将传统的单视角训练范式转变为多视角训练策略。通过我们提出的多视角调节，3D高斯特性得到进一步优化，避免过度拟合某些训练视角。作为一种通用解决方案，我们在各种场景和不同高斯变体中提高了整体准确性。2）受到额外视角带来的好处启发，我们进一步提出了一种交叉内在引导方案，引导进行不同分辨率的由粗到细的训练过程。3）在我们的多视角调节训练基础上，我们进一步提出了一种交叉射线致密化策略，从一组视角中在射线相交区域密集更多的高斯核。4）通过进一步研究致密化策略，我们发现当某些视角差异显著时，致密化效果应该得到增强。作为解决方案，我们提出了一种新颖的多视角增强致密化策略，鼓励3D高斯函数根据需要致密化到足够数量，从而提高重建准确性。

English

Recent works in volume rendering, e.g. NeRF and 3D Gaussian Splatting (3DGS), significantly advance the rendering quality and efficiency with the help of the learned implicit neural radiance field or 3D Gaussians. Rendering on top of an explicit representation, the vanilla 3DGS and its variants deliver real-time efficiency by optimizing the parametric model with single-view supervision per iteration during training which is adopted from NeRF. Consequently, certain views are overfitted, leading to unsatisfying appearance in novel-view synthesis and imprecise 3D geometries. To solve aforementioned problems, we propose a new 3DGS optimization method embodying four key novel contributions: 1) We transform the conventional single-view training paradigm into a multi-view training strategy. With our proposed multi-view regulation, 3D Gaussian attributes are further optimized without overfitting certain training views. As a general solution, we improve the overall accuracy in a variety of scenarios and different Gaussian variants. 2) Inspired by the benefit introduced by additional views, we further propose a cross-intrinsic guidance scheme, leading to a coarse-to-fine training procedure concerning different resolutions. 3) Built on top of our multi-view regulated training, we further propose a cross-ray densification strategy, densifying more Gaussian kernels in the ray-intersect regions from a selection of views. 4) By further investigating the densification strategy, we found that the effect of densification should be enhanced when certain views are distinct dramatically. As a solution, we propose a novel multi-view augmented densification strategy, where 3D Gaussians are encouraged to get densified to a sufficient number accordingly, resulting in improved reconstruction accuracy.