PerceptionDLM: 基於多模態擴散語言模型的並行區域感知

摘要

多模态大型語言模型（MLLMs）在視覺理解任務中已取得顯著進展。然而，現有大多數MLLMs依賴於自回歸生成，這限制了其在需要對多個區域進行描述（captioning）的感知任務中的效率。在本研究中，我們提出 PerceptionDLM，這是一種針對高效並行區域感知進行優化的多模態擴散語言模型。PerceptionDLM 基於 PerceptionDLM-Base 構建，後者是一個強大的基礎基準模型，在開源的擴散式 MLLMs 中達到了最先進的性能。我們的架構充分發揮了 DLM 的並行解碼特性。具體而言，我們引入了高效的提示（prompting）技術和結構化注意力遮罩，以實現對多個遮罩區域的同時感知，從而使模型能夠在序列和 token 層面上並行生成區域描述。與現有依次處理區域的方法相比，此設計顯著提升了推理效率。為了系統性地評估 DLM 在視覺感知能力上的並行特性，我們通過擴展 DLC-Bench，為每張圖像納入多個區域遮罩，建構了一個全新的並行詳細局部描寫基準（ParaDLC-Bench），從而實現對描寫品質與推理效率的聯合評估。實驗結果表明，PerceptionDLM 在保持區域描寫競爭力性能的同時，在多區域感知任務中實現了顯著的速度提升。我們的結果凸顯了多模態擴散語言模型在高效並行視覺感知方面的潛力。據我們所知，這是首次利用擴散語言模型的優勢實現並行區域描述與感知的工作。我們已開源相關的程式碼、模型與資料集。

English

Multimodal large language models (MLLMs) have achieved remarkable progress in visual understanding tasks. However, most existing MLLMs rely on autoregressive generation, which limits their efficiency for perception tasks that require captioning multiple regions. In this work, we propose PerceptionDLM, a multimodal diffusion language model optimized for efficient parallel region perception. Built upon PerceptionDLM-Base, a strong foundational baseline that achieves state-of-the-art performance among open-source diffusion MLLMs, our architecture fully leverages the parallel decoding nature of DLMs. Specifically, we introduce efficient prompting and structured attention masking to enable simultaneous perception of multiple masked regions, allowing the model to generate region descriptions in parallel at both the sequence and token levels. This design significantly improves inference efficiency compared with existing approaches that process regions sequentially. To systematically evaluate the parallelism property of visual perception capability for DLMs, we construct a new Parallel Detailed Localized Captioning Benchmark (ParaDLC-Bench) by scaling the DLC-Bench to include multiple region masks per image, enabling joint evaluation of both caption quality and inference efficiency. Experiments demonstrate that PerceptionDLM maintains competitive performance in region captioning while achieving substantial speed improvements for multi-region perception tasks. Our results highlight the potential of multimodal diffusion language models for efficient, parallel visual perception. To the best of our knowledge, we are the first to achieve parallel region caption and perception by leveraging the advantages of diffusion language models. Code, models, and datasets are released.