并非所有去噪步骤都同等重要：面向快速掩码扩散语言模型的分阶段调度策略

摘要

近期掩码扩散语言模型（MDLM）的发展使其与自回归语言模型的质量差距逐渐缩小，但由于生成过程需借助大型Transformer进行多次全序列去噪迭代，且无法像自回归解码那样受益于KV缓存，其采样成本依然高昂。本研究利用扩散框架的灵活性，探索模型调度策略——在部分去噪步骤中使用小型MDLM替代完整模型。基于OpenWebText和LM1B数据集训练的模型实验表明，相较于中间阶段，扩散过程的早期与晚期步骤对此类替换具有显著鲁棒性。在无条件生成与前缀条件生成任务中，该策略能以仅小幅增加生成困惑度为代价，实现高达17%的浮点运算量削减，同时保持样本多样性。我们通过基于时间步的损失函数分析、大小模型间KL散度评估以及粗粒度步骤段的穷举搜索，验证了扩散轨迹中段具有跨数据集一致的最高敏感性。这些发现表明，无需改变模型架构的简易调度规则即可显著加速MDLM采样，同时基本保持生成质量。

English

Recent advances in masked diffusion language models (MDLMs) narrow the quality gap to autoregressive LMs, but their sampling remains expensive because generation requires many full-sequence denoising passes with a large Transformer and, unlike autoregressive decoding, cannot benefit from KV caching. In this work, we exploit the flexibility of the diffusion framework and study model scheduling, where a smaller MDLM replaces the full model at a subset of denoising steps. Across models trained on OpenWebText and LM1B, we show that early and late denoising steps are substantially more robust to such replacement than middle steps, enabling up to a 17% reduction in FLOPs with only modest degradation in generative perplexity under both unconditional and prefix-conditional generation, while preserving sample diversity. We support these findings with a step-importance analysis based on loss and KL divergence between small and large models across timesteps, as well as an exhaustive search over coarse step segments, both of which identify the middle of the diffusion trajectory as most sensitive consistently across datasets. Our results suggest that simple, architecture-agnostic scheduling rules can significantly accelerate MDLM sampling while largely preserving generation quality.

并非所有去噪步骤都同等重要：面向快速掩码扩散语言模型的分阶段调度策略

Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models

摘要

Support