擴散語言模型研究綜述

A Survey on Diffusion Language Models

August 14, 2025

作者: Tianyi Li, Mingda Chen, Bowei Guo, Zhiqiang Shen

cs.AI

摘要

擴散語言模型（Diffusion Language Models, DLMs）正迅速崛起，成為主導的自迴歸（Autoregressive, AR）範式之外一個強大且極具潛力的替代方案。通過迭代去噪過程並行生成詞元，DLMs在降低推理延遲和捕捉雙向上下文方面具有內在優勢，從而實現對生成過程的精細控制。在實現數倍速度提升的同時，最新進展已使DLMs展現出與自迴歸模型相媲美的性能，使其成為各種自然語言處理任務中的一個引人注目的選擇。在本綜述中，我們全面概述了當前DLM的發展現狀。我們追溯了其演變歷程及其與其他範式（如自迴歸和掩碼語言模型）的關係，並涵蓋了基礎原理和最先進的模型。我們的工作提供了一個最新、全面的分類體系，並對當前技術進行了深入分析，從預訓練策略到先進的後訓練方法。本綜述的另一貢獻是對DLM推理策略和優化技術的全面回顧，包括解碼並行性、緩存機制和生成質量的改進。我們還重點介紹了DLM在多模態擴展方面的最新方法，並描繪了其在各種實際場景中的應用。此外，我們的討論涉及DLM的局限性和挑戰，包括效率、長序列處理和基礎設施需求，同時勾勒出未來的研究方向，以維持這一快速發展領域的進步。項目GitHub地址為https://github.com/VILA-Lab/Awesome-DLMs。

English

Diffusion Language Models (DLMs) are rapidly emerging as a powerful and promising alternative to the dominant autoregressive (AR) paradigm. By generating tokens in parallel through an iterative denoising process, DLMs possess inherent advantages in reducing inference latency and capturing bidirectional context, thereby enabling fine-grained control over the generation process. While achieving a several-fold speed-up, recent advancements have allowed DLMs to show performance comparable to their autoregressive counterparts, making them a compelling choice for various natural language processing tasks. In this survey, we provide a holistic overview of the current DLM landscape. We trace its evolution and relationship with other paradigms, such as autoregressive and masked language models, and cover both foundational principles and state-of-the-art models. Our work offers an up-to-date, comprehensive taxonomy and an in-depth analysis of current techniques, from pre-training strategies to advanced post-training methods. Another contribution of this survey is a thorough review of DLM inference strategies and optimizations, including improvements in decoding parallelism, caching mechanisms, and generation quality. We also highlight the latest approaches to multimodal extensions of DLMs and delineate their applications across various practical scenarios. Furthermore, our discussion addresses the limitations and challenges of DLMs, including efficiency, long-sequence handling, and infrastructure requirements, while outlining future research directions to sustain progress in this rapidly evolving field. Project GitHub is available at https://github.com/VILA-Lab/Awesome-DLMs.