离散扩散的漏洞利用：确定性绕过采样壁垒

摘要

离散扩散模型通过并行解码为自回归生成提供了有前景的替代方案，但其存在采样壁垒问题：一旦进行类别采样，丰富的分布信息就会坍缩为独热向量而无法跨步传播，迫使后续步骤只能在有限信息下运行。为缓解此问题，我们提出"潜径迂回"机制——通过确定性潜路径保留分布信息的新颖而简洁的方法，由此构建潜径迂回离散扩散模型（LDDM）。采用自条件策略高效训练后，LDDM实现了显著提升：生成困惑度较现有基线最高降低61%，缩小（部分任务甚至反超）与自回归模型的差距，并生成更连贯的文本。在推理任务中，LDDM于Countdown和24点游戏等算术基准测试上也表现更优。这些结果同时表明，潜径迂回机制能有效缓解空闲步数与振荡现象，为高质量非自回归文本生成提供了可扩展路径。

English

Discrete diffusion models offer a promising alternative to autoregressive generation through parallel decoding, but they suffer from a sampling wall: once categorical sampling occurs, rich distributional information collapses into one-hot vectors and cannot be propagated across steps, forcing subsequent steps to operate with limited information. To mitigate this problem, we introduce Loopholing, a novel and simple mechanism that preserves this information via a deterministic latent pathway, leading to Loopholing Discrete Diffusion Models (LDDMs). Trained efficiently with a self-conditioning strategy, LDDMs achieve substantial gains-reducing generative perplexity by up to 61% over prior baselines, closing (and in some cases surpassing) the gap with autoregressive models, and producing more coherent text. Applied to reasoning tasks, LDDMs also improve performance on arithmetic benchmarks such as Countdown and Game of 24. These results also indicate that loopholing mitigates idle steps and oscillations, providing a scalable path toward high-quality non-autoregressive text generation.

离散扩散的漏洞利用：确定性绕过采样壁垒

Loopholing Discrete Diffusion: Deterministic Bypass of the Sampling Wall

摘要

Support