高斯物件：僅需四張圖像即可使用高斯飛濺獲得高質量3D物件

GaussianObject: Just Taking Four Images to Get A High-Quality 3D Object with Gaussian Splatting

February 15, 2024

作者: Chen Yang, Sikuang Li, Jiemin Fang, Ruofan Liang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, Qi Tian

cs.AI

摘要

重建和渲染來自高度稀疏視圖的3D物體對於推動3D視覺技術應用並改善使用者體驗至關重要。然而，來自稀疏視圖的影像僅包含非常有限的3D信息，導致兩個重要挑戰：1）由於匹配的影像太少，難以建立多視圖一致性；2）部分遺漏或高度壓縮的物體信息，因為視圖覆蓋不足。為應對這些挑戰，我們提出了一個名為GaussianObject的框架，使用高斯點陣來表示和渲染3D物體，僅需4個輸入影像即可實現高質量渲染。我們首先引入了視覺外殼和浮動消除技術，明確地將結構先驗注入到初始優化過程中，以幫助建立多視圖一致性，從而產生粗糙的3D高斯表示。然後，我們基於擴散模型構建了一個高斯修復模型，以補充遺漏的物體信息，進一步優化高斯。我們設計了一個自生成策略，用於獲取訓練修復模型的影像對。我們的GaussianObject在幾個具有挑戰性的數據集上進行了評估，包括MipNeRF360、OmniObject3D和OpenIllumination，僅從4個視圖中實現了強大的重建結果，並顯著優於先前的最先進方法。

English

Reconstructing and rendering 3D objects from highly sparse views is of critical importance for promoting applications of 3D vision techniques and improving user experience. However, images from sparse views only contain very limited 3D information, leading to two significant challenges: 1) Difficulty in building multi-view consistency as images for matching are too few; 2) Partially omitted or highly compressed object information as view coverage is insufficient. To tackle these challenges, we propose GaussianObject, a framework to represent and render the 3D object with Gaussian splatting, that achieves high rendering quality with only 4 input images. We first introduce techniques of visual hull and floater elimination which explicitly inject structure priors into the initial optimization process for helping build multi-view consistency, yielding a coarse 3D Gaussian representation. Then we construct a Gaussian repair model based on diffusion models to supplement the omitted object information, where Gaussians are further refined. We design a self-generating strategy to obtain image pairs for training the repair model. Our GaussianObject is evaluated on several challenging datasets, including MipNeRF360, OmniObject3D, and OpenIllumination, achieving strong reconstruction results from only 4 views and significantly outperforming previous state-of-the-art methods.