離散擴散模型：從標記化到生成的統一框架

摘要

離散去噪擴散模型（DDMs）近期已成為離散數據建模中自回歸（AR）模型的引人注目替代方案，具備並行生成與迭代式全域修正能力。與連續擴散（其狀態空間固定）不同，DDMs 的設計本質上取決於離散狀態空間的構建方式：包括分詞方案、詞彙拓撲結構及領域特定的結構化字母表。本研究提出一個統一的概念框架，透過底層離散狀態空間的建構來審視離散擴散模型。在此框架下，現有方法——如轉移矩陣、遮蔽/吸收態、以及分數/比值導向的模型——皆可視為共通設計空間中的不同實例。該框架進一步揭示訓練目標、推論演算法、擴展行為、系統優化及評估計畫之間的常見設計權衡，並為未來研究提出數個具前景的方向。

English

Discrete denoising diffusion models (DDMs) have recently emerged as a compelling alternative to autoregressive (AR) modeling for discrete data, offering parallel generation and iterative global refinement capabilities. Unlike continuous diffusion, where the state space is fixed, DDMs are fundamentally shaped by how the discrete state space is constructed: the tokenization scheme, the vocabulary topology, and domain-specific structural alphabets. This work introduces a unified conceptual framework that views discrete diffusion models through the construction of the underlying discrete state space. Within this framework, existing formulations, including transition-matrix, masking/absorbing-state, and score/ratio-based approaches, emerge as different instantiations of a common design space. The framework further exposes common design trade-offs across training objectives, inference algorithms, scaling behavior, systems optimization, and evaluation protocols, suggesting several promising directions for future research.