论离散性在扩散大语言模型中的作用

摘要

扩散模型为语言生成提供了诱人的特性，如并行解码和迭代优化，但文本的离散性和高度结构化特性对扩散原理的直接应用构成了挑战。本文从扩散过程和语言建模的视角重新审视扩散语言建模，并阐明了将扩散机制与语言特定需求区分开的五个特性。我们首先将现有方法归类为嵌入空间中的连续扩散和词元上的离散扩散，进而证明每种方法仅满足五项关键特性中的部分特性，因而反映了结构上的权衡。通过分析近期的大型扩散语言模型，我们发现了两个核心问题：（i）均匀噪声干扰未能考虑信息在文本位置间的分布规律；（ii）词元级边际训练无法在并行解码过程中捕捉多词元依赖关系。这些发现启示我们设计更贴合文本结构的扩散过程，并推动未来研究构建更具连贯性的扩散语言模型。

English

Diffusion models offer appealing properties for language generation, such as parallel decoding and iterative refinement, but the discrete and highly structured nature of text challenges the direct application of diffusion principles. In this paper, we revisit diffusion language modeling from the view of diffusion process and language modeling, and outline five properties that separate diffusion mechanics from language-specific requirements. We first categorize existing approaches into continuous diffusion in embedding space and discrete diffusion over tokens. We then show that each satisfies only part of the five essential properties and therefore reflects a structural trade-off. Through analyses of recent large diffusion language models, we identify two central issues: (i) uniform corruption does not respect how information is distributed across positions, and (ii) token-wise marginal training cannot capture multi-token dependencies during parallel decoding. These observations motivate diffusion processes that align more closely with the structure of text, and encourage future work toward more coherent diffusion language models.

论离散性在扩散大语言模型中的作用

On the Role of Discreteness in Diffusion LLMs

摘要

Support