基于视觉语言推理的城市社会语义分割

摘要

作为人类活动的枢纽，城市地表蕴含着丰富的语义实体。从卫星图像中分割这些多样化实体对一系列下游应用至关重要。当前先进的语义分割模型能够可靠地分割由物理属性定义的实体（如建筑物、水体），但在处理社会属性定义的类别（如学校、公园）时仍面临挑战。本研究通过视觉语言模型推理实现了社会语义分割。为此，我们推出了名为SocioSeg的城市社会语义分割数据集，该全新资源包含卫星影像、数字地图以及按层级结构组织的社会语义实体像素级标注。此外，我们提出名为SocioReasoner的创新视觉语言推理框架，通过跨模态识别与多阶段推理模拟人类识别标注社会语义实体的认知过程。我们采用强化学习优化这一不可微分流程，充分激发视觉语言模型的推理能力。实验表明，该方法在超越现有最优模型的同时展现出强大的零样本泛化性能。我们的数据集与代码已开源：https://github.com/AMAP-ML/SocioReasoner。

English

As hubs of human activity, urban surfaces consist of a wealth of semantic entities. Segmenting these various entities from satellite imagery is crucial for a range of downstream applications. Current advanced segmentation models can reliably segment entities defined by physical attributes (e.g., buildings, water bodies) but still struggle with socially defined categories (e.g., schools, parks). In this work, we achieve socio-semantic segmentation by vision-language model reasoning. To facilitate this, we introduce the Urban Socio-Semantic Segmentation dataset named SocioSeg, a new resource comprising satellite imagery, digital maps, and pixel-level labels of social semantic entities organized in a hierarchical structure. Additionally, we propose a novel vision-language reasoning framework called SocioReasoner that simulates the human process of identifying and annotating social semantic entities via cross-modal recognition and multi-stage reasoning. We employ reinforcement learning to optimize this non-differentiable process and elicit the reasoning capabilities of the vision-language model. Experiments demonstrate our approach's gains over state-of-the-art models and strong zero-shot generalization. Our dataset and code are available in https://github.com/AMAP-ML/SocioReasoner.

基于视觉语言推理的城市社会语义分割

Urban Socio-Semantic Segmentation with Vision-Language Reasoning

摘要

Support