论离散性在扩散大语言模型中的作用

摘要

扩散模型为语言生成提供了诱人的特性，如并行解码与迭代优化，但文本的离散性与高度结构化特性对直接应用扩散原理构成了挑战。本文从扩散过程与语言建模的双重视角重新审视扩散语言模型，归纳出区分扩散机制与语言特定需求的五项特性。我们首先将现有方法划分为嵌入空间的连续扩散和词元层面的离散扩散，进而证明每类方法仅能部分满足五项关键特性，反映出固有的结构权衡。通过对近期大型扩散语言模型的分析，我们揭示出两个核心问题：（i）均匀噪声干扰未能充分考虑信息在文本位置间的分布规律；（ii）词元边际训练无法捕捉并行解码过程中的多词元依赖关系。这些发现启示我们设计更贴合文本结构的扩散过程，推动未来研究构建更具连贯性的扩散语言模型。

English

Diffusion models offer appealing properties for language generation, such as parallel decoding and iterative refinement, but the discrete and highly structured nature of text challenges the direct application of diffusion principles. In this paper, we revisit diffusion language modeling from the view of diffusion process and language modeling, and outline five properties that separate diffusion mechanics from language-specific requirements. We first categorize existing approaches into continuous diffusion in embedding space and discrete diffusion over tokens. We then show that each satisfies only part of the five essential properties and therefore reflects a structural trade-off. Through analyses of recent large diffusion language models, we identify two central issues: (i) uniform corruption does not respect how information is distributed across positions, and (ii) token-wise marginal training cannot capture multi-token dependencies during parallel decoding. These observations motivate diffusion processes that align more closely with the structure of text, and encourage future work toward more coherent diffusion language models.

论离散性在扩散大语言模型中的作用

On the Role of Discreteness in Diffusion LLMs

摘要

Support