朝向逼真的基於實例的建模：透過3D高斯拼接

Towards Realistic Example-based Modeling via 3D Gaussian Stitching

August 28, 2024

作者: Xinyu Gao, Ziyi Yang, Bingchen Gong, Xiaoguang Han, Sipeng Yang, Xiaogang Jin

cs.AI

摘要

利用現有模型的部分來重新構建新模型，通常被稱為基於示例的建模，在計算機圖形領域是一種經典方法。先前的研究主要集中在形狀組合上，使它們很難用於從現實場景中捕獲的3D物體的逼真組合。這導致將多個 NeRFs 組合成單個3D場景，以實現無縫外觀混合。然而，目前的 SeamlessNeRF 方法由於其基於梯度的策略和基於網格的表示而難以實現對現實場景的互動編輯和和諧拼接。為此，我們提出了一種基於示例的建模方法，使用基於樣本引導合成的點基表示結合多個高斯場。具體來說，對於組合，我們創建了一個 GUI，可以實時分割和轉換多個場，輕鬆獲得由3D高斯飛濺（3DGS）表示的模型的語義有意義的組合。對於紋理混合，由於3DGS的離散和不規則性，直接應用梯度傳播如SeamlssNeRF並不支持。因此，提出了一種新的基於採樣的克隆方法，以在保留原始豐富紋理和內容的同時協調混合。我們的工作流程包括三個步驟：1）使用精心設計的 GUI 實時分割和轉換高斯模型，2）KNN 分析以識別源模型和目標模型之間交叉區域的邊界點，以及 3）使用基於採樣的克隆和梯度約束對目標模型進行兩階段優化。大量實驗結果驗證了我們的方法在逼真合成方面顯著優於先前的工作，展示了其實用性。更多演示可在 https://ingra14m.github.io/gs_stitching_website 上找到。

English

Using parts of existing models to rebuild new models, commonly termed as example-based modeling, is a classical methodology in the realm of computer graphics. Previous works mostly focus on shape composition, making them very hard to use for realistic composition of 3D objects captured from real-world scenes. This leads to combining multiple NeRFs into a single 3D scene to achieve seamless appearance blending. However, the current SeamlessNeRF method struggles to achieve interactive editing and harmonious stitching for real-world scenes due to its gradient-based strategy and grid-based representation. To this end, we present an example-based modeling method that combines multiple Gaussian fields in a point-based representation using sample-guided synthesis. Specifically, as for composition, we create a GUI to segment and transform multiple fields in real time, easily obtaining a semantically meaningful composition of models represented by 3D Gaussian Splatting (3DGS). For texture blending, due to the discrete and irregular nature of 3DGS, straightforwardly applying gradient propagation as SeamlssNeRF is not supported. Thus, a novel sampling-based cloning method is proposed to harmonize the blending while preserving the original rich texture and content. Our workflow consists of three steps: 1) real-time segmentation and transformation of a Gaussian model using a well-tailored GUI, 2) KNN analysis to identify boundary points in the intersecting area between the source and target models, and 3) two-phase optimization of the target model using sampling-based cloning and gradient constraints. Extensive experimental results validate that our approach significantly outperforms previous works in terms of realistic synthesis, demonstrating its practicality. More demos are available at https://ingra14m.github.io/gs_stitching_website.