LLaDA MoE V2：扩展混合专家扩散语言模型

摘要

扩散语言模型（dLLMs）为自回归（AR）语言建模提供了一种替代方案，然而混合专家（MoE）dLLMs的缩放行为仍未被充分理解。我们系统地刻画了优化超参数、算力分配和架构规模如何随MoE dLLMs扩展，并识别出与先前AR模型缩放趋势之间的定量差异。具体而言，在优化方面，最优名义批量大小随算力增长更快，而最优学习率随算力衰减更为迅速。在模型-数据分配方面，IsoFLOP分析揭示了轻微的数据侧倾斜：最优词元预算的增长速度快于激活的模型侧计算量。在MoE架构方面，在固定激活容量下，更大的规模日益倾向于更大的专家池，而适度的专家粒度始终保持有效，且分配给共享专家的激活容量比例在不同规模下保持稳定。在这些发现的指导下，我们从零开始在23.5T词元上训练了LLaDA MoE v2——一个30B-A3B的dLLM。该模型使用的预训练词元数量约为Qwen3的65%，却在多个知识、推理和代码基准上接近Qwen3。仅经过监督微调，它就在八个推理和代码基准中的七个上超越了SDAR Chat，并在若干任务上与Qwen3保持相近。这些结果为MoE dLLMs确立了实用的缩放定律和设计原则。

English

Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling, yet the scaling behavior of Mixture-of-Experts (MoE) dLLMs remains poorly understood. We systematically characterize how optimization hyperparameters, compute allocation, and architecture scale for MoE dLLMs, identifying quantitative differences from scaling trends previously reported for AR models. Specifically, for optimization, the optimal nominal batch size grows faster, while the optimal learning rate decays more rapidly with compute. For model--data allocation, IsoFLOP analysis reveals a slight data-side tilt: the optimal token budget grows faster than activated model-side computation. For MoE architecture, larger scales increasingly favor larger expert pools at fixed activated capacity, while moderate expert granularity remains consistently effective and the preferred fraction of activated capacity assigned to shared experts remains stable across scales. Guided by these findings, we train LLaDA MoE v2, a 30B-A3B dLLM, from scratch on 23.5T tokens. With approximately 65\% as many pretraining tokens as Qwen3, LLaDA MoE v2 approaches Qwen3 on several knowledge, reasoning, and coding benchmarks. After supervised fine-tuning alone, it outperforms SDAR Chat on seven of eight reasoning and coding benchmarks and remains close to Qwen3 on several tasks. These results establish practical scaling laws and design principles for MoE dLLMs.