LangFlow：连续扩散模型在语言建模领域比肩离散方法

LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling

April 15, 2026

作者: Yuxin Chen, Chumeng Liang, Hangke Sui, Ruihan Guo, Chaoran Cheng, Jiaxuan You, Ge Liu

cs.AI

摘要

连续扩散模型已成为图像等多种数据模态实现高保真度、可控性及少步生成的基础。然而在语言建模领域，由于数据空间的稀疏性及设计空间探索不足，现有连续扩散语言模型（DLM）的表现一直落后于离散模型。本研究通过将嵌入空间DLM与基于布雷格曼散度的流匹配方法相结合，并引入三项关键创新，成功弥合了这一差距：首先，我们推导出基于常微分方程的新型负对数似然边界，为连续流式语言模型提供理论评估依据；其次，提出信息均匀性原则来设定噪声调度，并据此设计基于冈贝尔分布的可学习噪声调度器；最后，通过引入自条件机制改进训练流程，发现其能显著提升嵌入空间DLM的似然度和生成质量，且效果与离散扩散模型存在本质差异。综合这些创新，LangFlow在困惑度（PPL）和生成困惑度（Gen. PPL）指标上均可媲美顶尖离散DLM，在LM1B数据集上达到30.0的PPL值，在OpenWebText数据集上达到24.6。在7个基准测试中，更有4个任务的零样本迁移表现超越自回归基线模型。LangFlow首次明确证明连续扩散是语言建模领域具有前景的研究范式。项目主页：https://github.com/nealchen2003/LangFlow

English

Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow