论Qwen3.8-Next架构的设计：评估、效率与训练稳定性

摘要

我们描述了Qwen3.8-Flash-Next的架构与消融实验。这是一个稀疏混合专家模型，拥有125B参数，每个token激活6B参数，另有51B参数的n-gram嵌入表存储在加速器之外。在十四个预训练基准上，该模型在八个基准上领先其前身397B-A17B，在其余基准上最多落后2.6个百分点，而激活参数仅为前者的1/3，训练token数为1/3，训练FLOPs约为1/9。Token混合采用逐层混合的Gated DeltaNet（GDN）与全局注意力，每四层中设置一层全注意力层；在继续预训练阶段，这些全注意力层被替换为Qwen稀疏注意力（QSA），后者通过压缩轻量级索引器以微块粒度对上下文进行评分。残差流被扩展为四个分支，并通过逐元素门控读取，我们将这种设计称为门控残差（Gated Residual，GR）。容量通过单个n-gram嵌入层在主干之外增加，该层的表从主机内存预取。我们沿三个维度评估每个候选改动：损失及下游基准；改动在训练、预填充和解码中的成本；以及其对最优超参数和训练稳定性的影响。损失与下游准确率并不总是同步变化：增大n-gram词表会单调降低损失，而下游准确率趋于饱和。该架构与Muon优化器共同将最优学习率和批大小上移，使得批大小预热不再必要，并显著提高了压力测试下的稳定性。损失、基准、效率和稳定性构成一个统一的设计问题。联合求解后，得到的方案同时更高效、能力更强且更稳定。

English

We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and additional 51B parameters of n-gram embedding tables held off the accelerator. On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs. Token mixing uses a layer-wise hybrid of Gated DeltaNet (GDN) and global attention, with one full-attention layer in every four; at continued-pretraining time those full-attention layers are replaced by Qwen Sparse Attention (QSA), which scores context at micro-block granularity with a compressed lightweight indexer. The residual stream is widened to four branches and read through an elementwise gate, a design we call the Gated Residual (GR). Capacity is added outside the backbone by a single n-gram embedding layer whose tables are prefetched from host memory. We evaluate every candidate change along three axes: loss together with downstream benchmarks; the cost of the change in training, prefill and decode; and its effect on the optimal hyperparameters and training stability. Loss and downstream accuracy do not always move together: enlarging the n-gram vocabulary lowers loss monotonically while downstream accuracy saturates. The architecture and the Muon optimizer together shift the optimal learning rate and batch size upwards, render batch-size warmup unnecessary, and substantially improve stability under stress tests. Loss, benchmarks, efficiency and stability form one design problem. Solved jointly, they yield a recipe that is simultaneously more efficient, more capable and more stable.