SegEarth-OV3：探索SAM 3在遙感影像開放詞彙語義分割中的應用

摘要

目前多數免訓練的開放詞彙語義分割方法均基於CLIP架構。儘管這些方法已取得一定進展，但在精確定位方面仍面臨挑戰，或需透過複雜流程整合多個獨立模組，尤其在充斥大量密集小目標的遙感場景中更為明顯。近期提出的Segment Anything Model 3（SAM 3）將分割與識別功能統一於可提示式框架中。本文針對無需訓練的遙感開放詞彙語義分割任務，開展了SAM 3應用的初步探索。首先，我們設計了掩碼融合策略，合併SAM 3語義分割頭與Transformer解碼器（實例頭）的輸出，藉此融合兩類分割頭的優勢以提升地表覆蓋識別效果。其次，利用存在性評分頭輸出的存在分數過濾場景中不存在的類別，從而降低地理空間場景中因龐大詞彙量與像素級處理所導致的誤報。我們在多個遙感數據集上進行評估，實驗表明這種簡易改編能實現優異性能，展現了SAM 3在遙感開放詞彙語義分割領域的應用潛力。相關代碼已開源於：https://github.com/earth-insights/SegEarth-OV-3。

English

Most existing methods for training-free Open-Vocabulary Semantic Segmentation (OVSS) are based on CLIP. While these approaches have made progress, they often face challenges in precise localization or require complex pipelines to combine separate modules, especially in remote sensing scenarios where numerous dense and small targets are present. Recently, Segment Anything Model 3 (SAM 3) was proposed, unifying segmentation and recognition in a promptable framework. In this paper, we present a preliminary exploration of applying SAM 3 to the remote sensing OVSS task without any training. First, we implement a mask fusion strategy that combines the outputs from SAM 3's semantic segmentation head and the Transformer decoder (instance head). This allows us to leverage the strengths of both heads for better land coverage. Second, we utilize the presence score from the presence head to filter out categories that do not exist in the scene, reducing false positives caused by the vast vocabulary sizes and patch-level processing in geospatial scenes. We evaluate our method on extensive remote sensing datasets. Experiments show that this simple adaptation achieves promising performance, demonstrating the potential of SAM 3 for remote sensing OVSS. Our code is released at https://github.com/earth-insights/SegEarth-OV-3.

SegEarth-OV3：探索SAM 3在遙感影像開放詞彙語義分割中的應用

SegEarth-OV3: Exploring SAM 3 for Open-Vocabulary Semantic Segmentation in Remote Sensing Images

摘要

Support