更好、更强、更快、更广：面向MLLM分割的结构化全掩码预测

摘要

基于MLLM的分割面临一个核心分割三难困境：高分割性能、保持对话能力与快速推理难以兼得。嵌入预测方法可能通过像素级目标干扰语言建模，而逐词生成对于稠密掩码而言效率低下。我们提出全掩码预测（All-Mask Prediction），将自回归对话与非自回归掩码预测解耦。其二元实例化方法STAMP（Simultaneous Textual All-Mask Prediction）在词汇表内发出<SEG>触发标记，将图像对齐的掩码令牌与对应补丁特征融合，并利用混合注意力在一次前向过程中将所有令牌分类为前景或背景。由此，该方法在保持多模态能力与高效推理的同时，实现了强指代分割与推理分割。然而，二元掩码无法在不进行重复目标特定预测的情况下保留多个语义或实例身份。为此，我们提出结构化全掩码预测（Structured All-Mask Prediction）并开发了STAMPlus。该方法生成带有显式ID和可选边界框的目标列表，将这些ID绑定到共享的多类掩码空间，并在一次非自回归前向过程中联合预测所有目标。一个统一的检查点保留了STAMP的指代与推理能力，同时扩展至开放词汇语义分割、实例感知分割以及遥感小目标分割，其中高分辨率掩码令牌缩放保留了更精细的空间证据。在上述各场景中，STAMPlus实现了最先进的分割性能，保持了通用多模态指令跟随能力，并将12类别的推理延迟从STAMP重复推理的13.50秒降低至5.16秒。进一步的分析表明，准确的目标线索能够提升分割性能，而学习到的空间接地有助于二次查看推理。总体而言，STAMPlus在超越单目标预测的范畴上解决了两难困境。

English

MLLM-based segmentation faces a core segmentation trilemma: high segmentation performance, preserved dialogue ability, and fast inference. Embedding-prediction methods may disrupt language modeling through pixel-level objectives, whereas next-token generation is inefficient for dense masks. We propose All-Mask Prediction, decoupling autoregressive dialogue from non-autoregressive mask prediction. Its binary instantiation, STAMP (Simultaneous Textual All-Mask Prediction), emits an in-vocabulary <SEG> trigger, fuses image-aligned mask tokens with corresponding patch features, and uses hybrid attention to classify all tokens as foreground or background in one pass. It thereby combines strong referring and reasoning segmentation with preserved multimodal ability and efficient inference. However, binary masks cannot retain multiple semantic or instance identities without repeated target-specific predictions. We therefore propose Structured All-Mask Prediction and develop STAMPlus. It generates a target list with explicit IDs and optional boxes, binds these IDs to a shared multi-class mask space, and jointly predicts all targets in one non-autoregressive pass. A single unified checkpoint retains STAMP's referring and reasoning capabilities while extending to open-vocabulary semantic, instance-aware, and remote-sensing small-target segmentation, where high-resolution mask-token scaling preserves finer spatial evidence. Across these settings, STAMPlus achieves state-of-the-art segmentation performance, preserves general multimodal instruction following, and reduces 12-category latency from 13.50s for repeated STAMP inference to 5.16s. Further analyses show that accurate target cues improve segmentation and learned spatial grounding benefits look-twice reasoning. Overall, STAMPlus resolves the trilemma beyond single-target prediction.