SAMPart3D：在3D对象中分割任意部分

SAMPart3D: Segment Any Part in 3D Objects

November 11, 2024

作者: Yunhan Yang, Yukun Huang, Yuan-Chen Guo, Liangjun Lu, Xiaoyang Wu, Edmund Y. Lam, Yan-Pei Cao, Xihui Liu

cs.AI

摘要

3D部件分割是三维感知中至关重要且具有挑战性的任务，在机器人技术、三维生成和三维编辑等应用中发挥着关键作用。最近的方法利用强大的视觉语言模型（VLMs）进行二维到三维知识蒸馏，实现了零样本三维部件分割。然而，这些方法受制于对文本提示的依赖，限制了在大规模未标记数据集上的可扩展性以及处理部件模糊性的灵活性。在这项工作中，我们引入了SAMPart3D，一种可扩展的零样本三维部件分割框架，可以将任何三维对象分割为多个粒度的语义部件，而无需预定义的部件标签集作为文本提示。为了实现可扩展性，我们使用文本无关的视觉基础模型来蒸馏三维特征提取骨干，从而实现对大规模未标记三维数据集的扩展以学习丰富的三维先验知识。为了实现灵活性，我们蒸馏出尺度条件下的部件感知三维特征，用于多个粒度的三维部件分割。一旦从尺度条件下的部件感知三维特征中获得分割部件，我们使用VLMs基于多视角渲染为每个部件分配语义标签。与以往方法相比，我们的SAMPart3D可以扩展到最新的大规模三维对象数据集Objaverse，并处理复杂的非常规对象。此外，我们提出了一个新的三维部件分割基准，以解决现有基准中对象和部件的缺乏多样性和复杂性问题。实验证明，我们的SAMPart3D明显优于现有的零样本三维部件分割方法，并可以促进各种应用，如部件级编辑和交互式分割。

English

3D part segmentation is a crucial and challenging task in 3D perception, playing a vital role in applications such as robotics, 3D generation, and 3D editing. Recent methods harness the powerful Vision Language Models (VLMs) for 2D-to-3D knowledge distillation, achieving zero-shot 3D part segmentation. However, these methods are limited by their reliance on text prompts, which restricts the scalability to large-scale unlabeled datasets and the flexibility in handling part ambiguities. In this work, we introduce SAMPart3D, a scalable zero-shot 3D part segmentation framework that segments any 3D object into semantic parts at multiple granularities, without requiring predefined part label sets as text prompts. For scalability, we use text-agnostic vision foundation models to distill a 3D feature extraction backbone, allowing scaling to large unlabeled 3D datasets to learn rich 3D priors. For flexibility, we distill scale-conditioned part-aware 3D features for 3D part segmentation at multiple granularities. Once the segmented parts are obtained from the scale-conditioned part-aware 3D features, we use VLMs to assign semantic labels to each part based on the multi-view renderings. Compared to previous methods, our SAMPart3D can scale to the recent large-scale 3D object dataset Objaverse and handle complex, non-ordinary objects. Additionally, we contribute a new 3D part segmentation benchmark to address the lack of diversity and complexity of objects and parts in existing benchmarks. Experiments show that our SAMPart3D significantly outperforms existing zero-shot 3D part segmentation methods, and can facilitate various applications such as part-level editing and interactive segmentation.