論Qwen3.8-Next架構設計：評估、效率與訓練穩定性

摘要

我們描述了 Qwen3.8-Flash-Next 的架構與消融研究。這是一個稀疏混合專家模型，擁有 125B 參數，每個詞元激活 6B 參數，並額外配備 51B 參數的 n-gram 嵌入表，儲存於加速器之外。在十四個預訓練基準測試中，該模型在八個項目上領先其 397B-A17B 的前身，在其餘項目上最多落後 2.6 分，同時僅使用三分之一的激活參數、三分之一的訓練詞元，以及約九分之一的訓練浮點運算量（FLOPs）。詞元混合採用了 Gated DeltaNet（GDN）與全域注意力的分層混合結構，每四層中有一層為全注意力層；在持續預訓練階段，這些全注意力層被替換為 Qwen 稀疏注意力（QSA），該機制以微塊粒度並透過壓縮的輕量級索引器對上下文進行評分。殘差流被擴展為四個分支，並透過元素級門控讀取，我們將此設計稱為門控殘差（Gated Residual, GR）。容量透過單一 n-gram 嵌入層在主幹網路之外增加，其查表從主機記憶體預取。我們沿三個維度評估每項候選變更：損失與下游基準測試；該變更在訓練、預填充（prefill）和解碼（decode）中的成本；以及其對最佳超參數與訓練穩定性的影響。損失與下游準確率並不總是同步變化：擴大 n-gram 詞彙表會單調地降低損失，但下游準確率會趨於飽和。該架構與 Muon 最佳化器共同將最佳學習率和批次大小向上調整，使批次大小預熱不再必要，並大幅提升了壓力測試下的穩定性。損失、基準測試、效率與穩定性構成單一設計問題。若能聯合求解，便能得到一套同時更具效率、能力更強且更穩定的方案。

English

We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and additional 51B parameters of n-gram embedding tables held off the accelerator. On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs. Token mixing uses a layer-wise hybrid of Gated DeltaNet (GDN) and global attention, with one full-attention layer in every four; at continued-pretraining time those full-attention layers are replaced by Qwen Sparse Attention (QSA), which scores context at micro-block granularity with a compressed lightweight indexer. The residual stream is widened to four branches and read through an elementwise gate, a design we call the Gated Residual (GR). Capacity is added outside the backbone by a single n-gram embedding layer whose tables are prefetched from host memory. We evaluate every candidate change along three axes: loss together with downstream benchmarks; the cost of the change in training, prefill and decode; and its effect on the optimal hyperparameters and training stability. Loss and downstream accuracy do not always move together: enlarging the n-gram vocabulary lowers loss monotonically while downstream accuracy saturates. The architecture and the Muon optimizer together shift the optimal learning rate and batch size upwards, render batch-size warmup unnecessary, and substantially improve stability under stress tests. Loss, benchmarks, efficiency and stability form one design problem. Solved jointly, they yield a recipe that is simultaneously more efficient, more capable and more stable.