高斯物体：仅需四幅图像即可使用高斯飞溅获得高质量的3D物体

GaussianObject: Just Taking Four Images to Get A High-Quality 3D Object with Gaussian Splatting

February 15, 2024

作者: Chen Yang, Sikuang Li, Jiemin Fang, Ruofan Liang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian

cs.AI

摘要

从高度稀疏视图重建和渲染3D对象对于推动3D视觉技术的应用和提升用户体验至关重要。然而，稀疏视图中的图像仅包含非常有限的3D信息，导致两个重要挑战：1）建立多视一致性困难，因为用于匹配的图像太少；2）部分遗漏或高度压缩的对象信息，因为视图覆盖不足。为了解决这些挑战，我们提出了高斯对象（GaussianObject）框架，用高斯飘点（Gaussian splatting）表示和渲染3D对象，仅使用4个输入图像即可实现高质量渲染。我们首先介绍了视觉外壳（visual hull）和浮动体消除（floater elimination）技术，明确将结构先验信息注入初始优化过程，帮助建立多视一致性，得到粗糙的3D高斯表示。然后，我们基于扩散模型构建了高斯修复模型，以补充遗漏的对象信息，进一步优化高斯。我们设计了自生成策略来获取用于训练修复模型的图像对。我们的高斯对象在几个具有挑战性的数据集上进行了评估，包括MipNeRF360、OmniObject3D和OpenIllumination，仅使用4个视图实现了强大的重建结果，并显著优于先前的最先进方法。

English

Reconstructing and rendering 3D objects from highly sparse views is of critical importance for promoting applications of 3D vision techniques and improving user experience. However, images from sparse views only contain very limited 3D information, leading to two significant challenges: 1) Difficulty in building multi-view consistency as images for matching are too few; 2) Partially omitted or highly compressed object information as view coverage is insufficient. To tackle these challenges, we propose GaussianObject, a framework to represent and render the 3D object with Gaussian splatting, that achieves high rendering quality with only 4 input images. We first introduce techniques of visual hull and floater elimination which explicitly inject structure priors into the initial optimization process for helping build multi-view consistency, yielding a coarse 3D Gaussian representation. Then we construct a Gaussian repair model based on diffusion models to supplement the omitted object information, where Gaussians are further refined. We design a self-generating strategy to obtain image pairs for training the repair model. Our GaussianObject is evaluated on several challenging datasets, including MipNeRF360, OmniObject3D, and OpenIllumination, achieving strong reconstruction results from only 4 views and significantly outperforming previous state-of-the-art methods.