突破硬性掩碼：擴散語言模型的漸進式詞元演化

摘要

擴散語言模型（DLMs）通過迭代優化實現平行解碼，為語言建模提供了極具前景的替代方案。然而，現有DLMs大多依賴硬二元遮罩與離散詞元分配，這既限制了早期決策的修正能力，也未能充分發揮中間概率表徵的潛力。本文提出EvoToken-DLM，一種創新型的基於擴散的語言建模方法，以演進式軟詞元分佈取代硬二元遮罩。該模型實現了從遮罩狀態到離散輸出的漸進式轉換，支援可修正的解碼過程。為有效支撐此演進機制，我們引入連續軌跡監督技術，使訓練目標與迭代概率更新保持一致。在多項基準測試中的廣泛實驗表明，EvoToken-DLM始終保持卓越性能，顯著優於現有的強擴散模型與遮罩DLM基線模型。項目頁面：https://aim-uofa.github.io/EvoTokenDLM。

English

Diffusion Language Models (DLMs) offer a promising alternative for language modeling by enabling parallel decoding through iterative refinement. However, most DLMs rely on hard binary masking and discrete token assignments, which hinder the revision of early decisions and underutilize intermediate probabilistic representations. In this paper, we propose EvoToken-DLM, a novel diffusion-based language modeling approach that replaces hard binary masks with evolving soft token distributions. EvoToken-DLM enables a progressive transition from masked states to discrete outputs, supporting revisable decoding. To effectively support this evolution, we introduce continuous trajectory supervision, which aligns training objectives with iterative probabilistic updates. Extensive experiments across multiple benchmarks show that EvoToken-DLM consistently achieves superior performance, outperforming strong diffusion-based and masked DLM baselines. Project webpage: https://aim-uofa.github.io/EvoTokenDLM.

突破硬性掩碼：擴散語言模型的漸進式詞元演化

Beyond Hard Masks: Progressive Token Evolution for Diffusion Language Models

摘要

Support