离散扩散模型：从标记化到生成的统一框架

摘要

离散去噪扩散模型（DDM）近期已成为离散数据自回归（AR）建模的一种引人注目的替代方案，具备并行生成和迭代全局细化能力。与状态空间固定的连续扩散不同，DDM 的形态根本上取决于离散状态空间的构建方式：包括分词方案、词汇拓扑以及领域特定的结构字母。本文提出了一个统一的概念框架，通过底层离散状态空间的构建来审视离散扩散模型。在该框架下，现有的公式（包括转移矩阵、掩码/吸收态以及分数/比率方法）均表现为共同设计空间中的不同实例。该框架进一步揭示了训练目标、推理算法、缩放行为、系统优化及评估协议中常见的设计权衡，为未来研究指出了若干有前景的方向。

English

Discrete denoising diffusion models (DDMs) have recently emerged as a compelling alternative to autoregressive (AR) modeling for discrete data, offering parallel generation and iterative global refinement capabilities. Unlike continuous diffusion, where the state space is fixed, DDMs are fundamentally shaped by how the discrete state space is constructed: the tokenization scheme, the vocabulary topology, and domain-specific structural alphabets. This work introduces a unified conceptual framework that views discrete diffusion models through the construction of the underlying discrete state space. Within this framework, existing formulations, including transition-matrix, masking/absorbing-state, and score/ratio-based approaches, emerge as different instantiations of a common design space. The framework further exposes common design trade-offs across training objectives, inference algorithms, scaling behavior, systems optimization, and evaluation protocols, suggesting several promising directions for future research.