基於視覺語言推理的都市社會語意分割

摘要

作為人類活動的樞紐，城市地表蘊含豐富的語義實體。從衛星影像中分割這些多樣化實體對諸多下游應用至關重要。當前先進的分割模型能可靠地分割由物理屬性定義的實體（如建築物、水體），但在社會定義的類別（如學校、公園）上仍存在困難。本研究通過視覺語言模型推理實現了社會語義分割。為此，我們推出名為SocioSeg的城市社會語義分割數據集，該新資源包含衛星影像、數位地圖以及按層次結構組織的社會語義實體像素級標註。此外，我們提出創新的視覺語言推理框架SocioReasoner，通過跨模態識別與多階段推理模擬人類識別與標註社會語義實體的過程。我們採用強化學習優化這一不可微分流程，激發視覺語言模型的推理能力。實驗證明，相較於現有頂尖模型，我們的方法具有顯著優勢，並展現出強大的零樣本泛化能力。數據集與代碼公開於：https://github.com/AMAP-ML/SocioReasoner。

English

As hubs of human activity, urban surfaces consist of a wealth of semantic entities. Segmenting these various entities from satellite imagery is crucial for a range of downstream applications. Current advanced segmentation models can reliably segment entities defined by physical attributes (e.g., buildings, water bodies) but still struggle with socially defined categories (e.g., schools, parks). In this work, we achieve socio-semantic segmentation by vision-language model reasoning. To facilitate this, we introduce the Urban Socio-Semantic Segmentation dataset named SocioSeg, a new resource comprising satellite imagery, digital maps, and pixel-level labels of social semantic entities organized in a hierarchical structure. Additionally, we propose a novel vision-language reasoning framework called SocioReasoner that simulates the human process of identifying and annotating social semantic entities via cross-modal recognition and multi-stage reasoning. We employ reinforcement learning to optimize this non-differentiable process and elicit the reasoning capabilities of the vision-language model. Experiments demonstrate our approach's gains over state-of-the-art models and strong zero-shot generalization. Our dataset and code are available in https://github.com/AMAP-ML/SocioReasoner.

基於視覺語言推理的都市社會語意分割

Urban Socio-Semantic Segmentation with Vision-Language Reasoning

摘要

Support