第123部分：從單視圖影像進行部位感知的3D重建

Part123: Part-aware 3D Reconstruction from a Single-view Image

May 27, 2024

作者: Anran Liu, Cheng Lin, Yuan Liu, Xiaoxiao Long, Zhiyang Dou, Hao-Xiang Guo, Ping Luo, Wenping Wang

cs.AI

摘要

最近，擴散模型的出現為單視角重建開辟了新的機會。然而，所有現有方法都將目標物體表示為一個缺乏任何結構信息的封閉網格，因此忽略了對於許多下游應用至關重要的基於部件的結構，重建形狀。此外，生成的網格通常存在較大的噪音、不平滑的表面和模糊的紋理，這使得使用3D分割技術獲得滿意的部分片段變得具有挑戰性。在本文中，我們提出了Part123，這是一個從單視角圖像進行部分感知的3D重建的新框架。我們首先使用擴散模型從給定圖像生成多視角一致的圖像，然後利用展示對於任意物體具有強大泛化能力的Segment Anything Model（SAM）生成多視角分割遮罩。為了有效地將2D基於部分的信息納入3D重建並處理不一致性，我們將對比學習引入神經渲染框架，基於多視角分割遮罩學習一個部分感知的特徵空間。同時還開發了基於聚類的算法，從重建模型中自動獲得3D部分分割結果。實驗表明，我們的方法能夠在各種物體上生成具有高質量分割部分的3D模型。與現有的非結構重建方法相比，我們方法生成的部分感知3D模型對於一些重要應用具有益處，包括特徵保留重建、基本拟合和3D形狀編輯。

English

Recently, the emergence of diffusion models has opened up new opportunities for single-view reconstruction. However, all the existing methods represent the target object as a closed mesh devoid of any structural information, thus neglecting the part-based structure, which is crucial for many downstream applications, of the reconstructed shape. Moreover, the generated meshes usually suffer from large noises, unsmooth surfaces, and blurry textures, making it challenging to obtain satisfactory part segments using 3D segmentation techniques. In this paper, we present Part123, a novel framework for part-aware 3D reconstruction from a single-view image. We first use diffusion models to generate multiview-consistent images from a given image, and then leverage Segment Anything Model (SAM), which demonstrates powerful generalization ability on arbitrary objects, to generate multiview segmentation masks. To effectively incorporate 2D part-based information into 3D reconstruction and handle inconsistency, we introduce contrastive learning into a neural rendering framework to learn a part-aware feature space based on the multiview segmentation masks. A clustering-based algorithm is also developed to automatically derive 3D part segmentation results from the reconstructed models. Experiments show that our method can generate 3D models with high-quality segmented parts on various objects. Compared to existing unstructured reconstruction methods, the part-aware 3D models from our method benefit some important applications, including feature-preserving reconstruction, primitive fitting, and 3D shape editing.