突破硬掩码：扩散语言模型的渐进式词元进化

摘要

扩散语言模型（DLM）通过迭代优化的并行解码机制，为语言建模提供了前景广阔的替代方案。然而，现有DLM大多依赖硬二元掩码和离散词元分配机制，这既阻碍了早期决策的修正，也未能充分利用中间概率表示。本文提出EvoToken-DLM这一创新性扩散语言建模方法，通过动态演化的软词元分布替代硬二元掩码。该模型实现了从掩码状态到离散输出的渐进式转换，支持可修正的解码过程。为有效支撑这种演化机制，我们引入连续轨迹监督技术，使训练目标与迭代概率更新保持对齐。在多基准测试上的广泛实验表明，EvoToken-DLM持续实现卓越性能，显著优于现有的扩散模型和掩码DLM基线方法。项目页面：https://aim-uofa.github.io/EvoTokenDLM。

English

Diffusion Language Models (DLMs) offer a promising alternative for language modeling by enabling parallel decoding through iterative refinement. However, most DLMs rely on hard binary masking and discrete token assignments, which hinder the revision of early decisions and underutilize intermediate probabilistic representations. In this paper, we propose EvoToken-DLM, a novel diffusion-based language modeling approach that replaces hard binary masks with evolving soft token distributions. EvoToken-DLM enables a progressive transition from masked states to discrete outputs, supporting revisable decoding. To effectively support this evolution, we introduce continuous trajectory supervision, which aligns training objectives with iterative probabilistic updates. Extensive experiments across multiple benchmarks show that EvoToken-DLM consistently achieves superior performance, outperforming strong diffusion-based and masked DLM baselines. Project webpage: https://aim-uofa.github.io/EvoTokenDLM.

突破硬掩码：扩散语言模型的渐进式词元进化

Beyond Hard Masks: Progressive Token Evolution for Diffusion Language Models

摘要

Support