扩散模型：通用分割学习器

Diffusion Model as a Generalist Segmentation Learner

April 27, 2026

作者: Haoxiao Wang, Antao Xiang, Haiyang Sun, Peilin Sun, Changhao Pan, Yifu Chen, Minjie Hong, Weijie Wang, Shuang Chen, Yue Chen, Zhou Zhao

cs.AI

摘要

扩散模型虽主要训练用于图像生成，但其去噪轨迹编码了丰富且空间对齐的视觉先验。本文论证了这些先验可应用于文本条件语义分割与开放词汇分割，并能泛化至多种下游任务，构建通用型扩散分割框架。具体而言，我们提出DiGSeg（作为通用分割学习器的扩散模型），将预训练扩散模型重构为统一分割框架。该方法将输入图像与真实标注掩码编码至潜空间，并拼接为扩散U-Net的条件信号。通过并行CLIP对齐的文本通路，在多尺度注入语言特征，使模型能将文本查询与动态演化的视觉表征对齐。这一设计将现成的扩散主干网络转化为通用接口，可基于外观特征与任意文本提示生成结构化分割掩码。大量实验表明，该方法在标准语义分割基准上达到领先性能，同时在开放词汇泛化及跨领域迁移（医疗、遥感、农业场景）中展现强大适应性——无需针对特定领域进行架构定制。这些结果表明，现代扩散主干网络可作为通用分割学习器而非纯生成器，显著缩小了视觉生成与视觉理解之间的鸿沟。

English

Diffusion models are primarily trained for image synthesis, yet their denoising trajectories encode rich, spatially aligned visual priors. In this paper, we demonstrate that these priors can be utilized for text-conditioned semantic and open-vocabulary segmentation, and this approach can be generalized to various downstream tasks to make a general-purpose diffusion segmentation framework. Concretely, we introduce DiGSeg (Diffusion Models as a Generalist Segmentation Learner), which repurposes a pretrained diffusion model into a unified segmentation framework. Our approach encodes the input image and ground-truth mask into the latent space and concatenates them as conditioning signals for the diffusion U-Net. A parallel CLIP-aligned text pathway injects language features across multiple scales, enabling the model to align textual queries with evolving visual representations. This design transforms an off-the-shelf diffusion backbone into a universal interface that produces structured segmentation masks conditioned on both appearance and arbitrary text prompts. Extensive experiments demonstrate state-of-the-art performance on standard semantic segmentation benchmarks, as well as strong open-vocabulary generalization and cross-domain transfer to medical, remote sensing, and agricultural scenarios-without domain-specific architectural customization. These results indicate that modern diffusion backbones can serve as generalist segmentation learners rather than pure generators, narrowing the gap between visual generation and visual understanding.