通过3D高斯拼接实现逼真的基于示例的建模

Towards Realistic Example-based Modeling via 3D Gaussian Stitching

August 28, 2024

作者: Xinyu Gao, Ziyi Yang, Bingchen Gong, Xiaoguang Han, Sipeng Yang, Xiaogang Jin

cs.AI

摘要

将现有模型的部分用于重建新模型，通常被称为基于示例的建模，在计算机图形领域是一种经典方法。先前的研究主要集中在形状组合上，使其在从现实场景中捕获的3D对象的逼真组合方面难以使用。这导致将多个NeRF合并为单个3D场景，以实现无缝外观混合。然而，当前的SeamlessNeRF方法由于其基于梯度的策略和基于网格的表示而难以实现对真实场景的交互式编辑和和谐拼接。为此，我们提出了一种基于示例的建模方法，使用样本引导合成将多个高斯场组合到基于点的表示中。具体而言，对于组合，我们创建了一个GUI，用于实时分割和转换多个场，轻松获得由3D高斯喷洒（3DGS）表示的模型的语义有意义的组合。对于纹理混合，由于3DGS的离散和不规则性质，直接应用梯度传播如SeamlessNeRF并不支持。因此，提出了一种新颖的基于采样的克隆方法，用于协调混合同时保留原始丰富的纹理和内容。我们的工作流程包括三个步骤：1）使用精心设计的GUI实时分割和转换高斯模型，2）KNN分析以识别源模型和目标模型之间交叉区域的边界点，以及3）使用基于采样的克隆和梯度约束对目标模型进行两阶段优化。广泛的实验结果验证了我们的方法在逼真合成方面明显优于先前的工作，展示了其实用性。更多演示可在https://ingra14m.github.io/gs_stitching_website找到。

English

Using parts of existing models to rebuild new models, commonly termed as example-based modeling, is a classical methodology in the realm of computer graphics. Previous works mostly focus on shape composition, making them very hard to use for realistic composition of 3D objects captured from real-world scenes. This leads to combining multiple NeRFs into a single 3D scene to achieve seamless appearance blending. However, the current SeamlessNeRF method struggles to achieve interactive editing and harmonious stitching for real-world scenes due to its gradient-based strategy and grid-based representation. To this end, we present an example-based modeling method that combines multiple Gaussian fields in a point-based representation using sample-guided synthesis. Specifically, as for composition, we create a GUI to segment and transform multiple fields in real time, easily obtaining a semantically meaningful composition of models represented by 3D Gaussian Splatting (3DGS). For texture blending, due to the discrete and irregular nature of 3DGS, straightforwardly applying gradient propagation as SeamlssNeRF is not supported. Thus, a novel sampling-based cloning method is proposed to harmonize the blending while preserving the original rich texture and content. Our workflow consists of three steps: 1) real-time segmentation and transformation of a Gaussian model using a well-tailored GUI, 2) KNN analysis to identify boundary points in the intersecting area between the source and target models, and 3) two-phase optimization of the target model using sampling-based cloning and gradient constraints. Extensive experimental results validate that our approach significantly outperforms previous works in terms of realistic synthesis, demonstrating its practicality. More demos are available at https://ingra14m.github.io/gs_stitching_website.