Nemotron-Labs-Diffusion：一种统一自回归、扩散和自推测解码的三模式语言模型

摘要

我们介绍了Nemotron-Labs-Diffusion，这是一种三模语言模型（LM），在单一架构中统一了自回归（AR）、扩散和自推测解码。通过联合的AR-扩散目标进行训练，Nemotron-Labs-Diffusion能够切换模式，以在不同部署环境和并发级别下维持高吞吐量。我们的研究显示：（1）AR和扩散目标具有互补性：扩散改进了前瞻规划能力，而AR提供了从左到右的语言先验信息。（2）在自推测模式下，扩散负责前瞻草拟，AR负责验证，这在接受率和实际设备效率上均优于多标记预测（MTP）方法。（3）光速分析进一步揭示了扩散的长期潜力，在最优采样器下，每次前向传递生成的token数比自推测模式多76.5%。通过扩展至3B、8B和14B参数规模，我们的Nemotron-Labs-Diffusion系列（包括基础模型、指令模型和视觉语言模型）在准确性和速度上均持续优于最先进的开源AR和扩散LM。例如，Nemotron-Labs-Diffusion-8B每次前向传递解码的token数比Qwen3-8B多6倍，且准确性相当，在GB200 GPU上使用SGLang运行SPEED-Bench时，吞吐量提升了4倍。

English

We introduce Nemotron-Labs-Diffusion, a tri-mode language model (LM) that unifies AR, diffusion, and self-speculation decoding within a single architecture. Trained with a joint AR-diffusion objective, Nemotron-Labs-Diffusion can switch modes to sustain high throughput across deployment settings and concurrency levels. Our study shows that (1) AR and diffusion objectives are complementary: diffusion improves lookahead planning, while AR provides left-to-right linguistic priors. (2) In self-speculation mode, diffusion drafts while AR verifies, outperforming multi-token prediction (MTP) methods in both acceptance rate and real-device efficiency. (3) A speed-of-light analysis further demonstrates diffusion's long-term potential, with up to 76.5% more tokens per forward pass than self-speculation under an optimal sampler. Scaling to 3B, 8B, and 14B parameters, our Nemotron-Labs-Diffusion family, including base, instruct, and vision-language models, consistently outperforms state-of-the-art open-source AR and diffusion LMs in both accuracy and speed. For example, Nemotron-Labs-Diffusion-8B decodes 6x more tokens per forward than Qwen3-8B with comparable accuracy, translating to 4x higher throughput on SPEED-Bench with SGLang on a GB200 GPU.