从逐词生成到逐块生成：扩散大语言模型的合理适配路径

From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs

December 7, 2025

作者: Yuchuan Tian, Yuchen Liang, Jiacheng Sun, Shuo Zhang, Guangwen Yang, Yingte Shu, Sibo Fang, Tianyu Guo, Kai Han, Chao Xu, Hanting Chen, Xinghao Chen, Yunhe Wang

cs.AI

摘要

大型语言模型（LLMs）在生成任务上表现出色，但主流的自回归解码方式存在固有串行性，形成吞吐量瓶颈。扩散语言模型（DLMs）——尤其是分块变体——支持并行生成与块内双向推理，然而从头训练大型DLMs成本高昂，且浪费成熟自回归模型的知识储备。先前"适配"尝试或通过修改逻辑值/随机扩展注意力掩码实现全序列扩散，或简单将自回归权重移植至块扩散框架，均未解决自回归因果性与块双向性的根本矛盾。我们通过将自回归视为块大小=1的块扩散模型，将适配重构为自回归到块扩散的范式内路径转换。具体而言，我们设计如下适配路径：采用上下文因果注意力掩码（上下文保持因果性，仅在当前激活块内双向）、高效并行适配流程、辅助自回归损失以最大化数据利用并保留预训练知识，以及逐步增加生成块大小。该方案可与掩码块扩散无缝集成，并保持训练-推理一致性。基于这些组件，NBDiff-7B（基础版与指令版）能够继承长上下文建模与推理能力，在7B级DLMs中实现最优性能，在通用知识、数学和代码基准测试上较基线模型取得显著提升。这些结果表明，基于原理的自回归到块扩散适配是替代从头训练DLMs的高效计算方案。代码地址：https://github.com/YuchuanTian/NBDiff。

English

Large language models (LLMs) excel at generation but dominant autoregressive (AR) decoding is inherently sequential, creating a throughput bottleneck. Diffusion Language Models (DLMs)--especially block-wise variants--enable parallel generation and intra-block bidirectional reasoning, yet training large DLMs from scratch is costly and wastes the knowledge in mature AR checkpoints. Prior "adaptation" attempts either modify logits or randomly grow attention masks to full-sequence diffusion, or simply transplant AR weights into a block-diffusion recipe, leaving a fundamental mismatch between AR causality and block-wise bidirectionality unaddressed. We reframe adaptation as a intra-paradigm path from AR to Block-Diffusion by viewing AR as Block-Diffusion with blocksize=1. Concretely, we design the pathway of adaptation as follows: we use a context-causal attention mask (causal in context, bidirectional only within the active block), an efficient parallel adaptation procedure, an auxiliary AR loss to maximize data utilization and retain pretrained knowledge, and gradual increment of the generation block size. The recipe integrates cleanly with masked block-diffusion and maintains train-inference consistency. Built on these components, NBDiff-7B (Base and Instruct) could inherit the long-context modeling and reasoning capabilities, and achieve state-of-the-art performance among the 7B-class DLMs, delivering strong gains on general-knowledge, math, and code benchmarks over strong baselines. These results demonstrate that principled AR-to-block-diffusion adaptation is an effective and compute-efficient alternative to training DLMs from scratch. Codes: https://github.com/YuchuanTian/NBDiff.