第123部分：从单视图图像中感知部件的三维重建

Part123: Part-aware 3D Reconstruction from a Single-view Image

May 27, 2024

作者: Anran Liu, Cheng Lin, Yuan Liu, Xiaoxiao Long, Zhiyang Dou, Hao-Xiang Guo, Ping Luo, Wenping Wang

cs.AI

摘要

最近，扩散模型的出现为单视图重建开辟了新的机遇。然而，所有现有方法都将目标对象表示为一个缺乏任何结构信息的封闭网格，因此忽略了对重建形状的许多下游应用至关重要的基于部件的结构。此外，生成的网格通常存在大量噪音、不平滑的表面和模糊的纹理，使得使用3D分割技术获得满意的部分分割变得具有挑战性。在本文中，我们提出了Part123，这是一个从单视图图像进行部分感知3D重建的新框架。我们首先使用扩散模型从给定图像生成多视图一致的图像，然后利用“任意分割模型”（SAM），该模型展示了对任意对象具有强大泛化能力，生成多视图分割掩模。为了有效地将2D基于部件的信息纳入3D重建并处理不一致性，我们将对比学习引入到神经渲染框架中，基于多视图分割掩模学习部分感知特征空间。还开发了基于聚类的算法，可以自动从重建模型中导出3D部分分割结果。实验证明，我们的方法能够在各种对象上生成具有高质量分割部分的3D模型。与现有的非结构化重建方法相比，我们方法生成的部分感知3D模型有利于一些重要应用，包括特征保留重建、基本拟合和3D形状编辑。

English

Recently, the emergence of diffusion models has opened up new opportunities for single-view reconstruction. However, all the existing methods represent the target object as a closed mesh devoid of any structural information, thus neglecting the part-based structure, which is crucial for many downstream applications, of the reconstructed shape. Moreover, the generated meshes usually suffer from large noises, unsmooth surfaces, and blurry textures, making it challenging to obtain satisfactory part segments using 3D segmentation techniques. In this paper, we present Part123, a novel framework for part-aware 3D reconstruction from a single-view image. We first use diffusion models to generate multiview-consistent images from a given image, and then leverage Segment Anything Model (SAM), which demonstrates powerful generalization ability on arbitrary objects, to generate multiview segmentation masks. To effectively incorporate 2D part-based information into 3D reconstruction and handle inconsistency, we introduce contrastive learning into a neural rendering framework to learn a part-aware feature space based on the multiview segmentation masks. A clustering-based algorithm is also developed to automatically derive 3D part segmentation results from the reconstructed models. Experiments show that our method can generate 3D models with high-quality segmented parts on various objects. Compared to existing unstructured reconstruction methods, the part-aware 3D models from our method benefit some important applications, including feature-preserving reconstruction, primitive fitting, and 3D shape editing.