基於視覺-語言模型的通用化少樣本3D點雲分割

Generalized Few-shot 3D Point Cloud Segmentation with Vision-Language Model

March 20, 2025

作者: Zhaochong An, Guolei Sun, Yun Liu, Runjia Li, Junlin Han, Ender Konukoglu, Serge Belongie

cs.AI

摘要

通用少樣本三維點雲分割（GFS-PCS）旨在使模型能夠利用少量支持樣本適應新類別，同時保持基礎類別的分割能力。現有的GFS-PCS方法通過與支持或查詢特徵的交互來增強原型，但仍受限於少樣本樣本所帶來的稀疏知識。與此同時，三維視覺語言模型（3D VLMs）能夠泛化至開放世界中的新類別，蘊含著豐富但帶有噪聲的新類別知識。在本研究中，我們提出了一種GFS-PCS框架，名為GFS-VL，它將來自3D VLMs的密集但帶噪聲的偽標籤與精確卻稀疏的少樣本樣本相結合，以最大化兩者的優勢。具體而言，我們提出了一種基於原型引導的偽標籤選擇方法，用於過濾低質量區域，隨後採用一種自適應填充策略，結合偽標籤上下文和少樣本樣本的知識，對過濾後的未標記區域進行自適應標註。此外，我們設計了一種新舊類別混合策略，將少樣本樣本嵌入訓練場景中，保留關鍵上下文以提升新類別的學習效果。鑑於當前GFS-PCS基準測試中多樣性的不足，我們引入了兩個包含多樣新類別的挑戰性基準，用於全面的泛化能力評估。實驗驗證了我們框架在不同模型和數據集上的有效性。我們的方法和基準測試為推動GFS-PCS在現實世界中的應用奠定了堅實基礎。代碼已開源於https://github.com/ZhaochongAn/GFS-VL。

English

Generalized few-shot 3D point cloud segmentation (GFS-PCS) adapts models to new classes with few support samples while retaining base class segmentation. Existing GFS-PCS methods enhance prototypes via interacting with support or query features but remain limited by sparse knowledge from few-shot samples. Meanwhile, 3D vision-language models (3D VLMs), generalizing across open-world novel classes, contain rich but noisy novel class knowledge. In this work, we introduce a GFS-PCS framework that synergizes dense but noisy pseudo-labels from 3D VLMs with precise yet sparse few-shot samples to maximize the strengths of both, named GFS-VL. Specifically, we present a prototype-guided pseudo-label selection to filter low-quality regions, followed by an adaptive infilling strategy that combines knowledge from pseudo-label contexts and few-shot samples to adaptively label the filtered, unlabeled areas. Additionally, we design a novel-base mix strategy to embed few-shot samples into training scenes, preserving essential context for improved novel class learning. Moreover, recognizing the limited diversity in current GFS-PCS benchmarks, we introduce two challenging benchmarks with diverse novel classes for comprehensive generalization evaluation. Experiments validate the effectiveness of our framework across models and datasets. Our approach and benchmarks provide a solid foundation for advancing GFS-PCS in the real world. The code is at https://github.com/ZhaochongAn/GFS-VL