為何門控DeltaNet能經受4位元量化：混合式270億參數大型語言模型中遞迴部分的NVFP4 W4A4

Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM

September 3, 2026

作者: Sergii Kozyrev, Davyd Maiboroda

cs.AI

摘要

混合大型語言模型（Hybrid LLMs）將 softmax attention 與線性注意力層（如 Gated DeltaNet, GDN）搭配使用，後者以固定大小的循環狀態總結上下文。早期社群對 Qwen3.8-27B 的 4-bit 量化（48 個 GDN 層、16 個 attention 層）將 GDN 區塊保留在 8 位元或 16 位元精度——尤其是其衰減（decay）與寫入強度（write-strength）閘門——背後的直覺是循環中的誤差會在長上下文上累積。我們透過構建 Minima 來檢驗該直覺：對全部 496 個線性層（含 GDN）採用 NVFP4 W4A4 量化。在 4K/32K 的困惑度、MMLU-Pro、GSM8K、AIME'25、GPQA-Diamond、LiveCodeBench 以及 RULER 至 64K 的檢索任務上，Minima 在種子雜訊範圍內與 BF16 相當（5 項任務平均 -0.52），同時是我們比較的方案中最小（17.5 GiB）且 prefill 最快（+14-19%）者；其 32K 困惑度差距亦隨位置縮小。一項由四部分組成的機制研究解釋了原因：(i) NVFP4 的 16 元素區塊縮放將殘差流中的極端離群值局部化，使各層角色的激活誤差趨於均等；(ii) 那些被認為脆弱的閘門投影反而是最不敏感的——softplus/exponential 與 sigmoid 參數化可將約 11% 的 GEMM 誤差壓縮為約 2% 的輸出誤差；(iii) delta 規則循環將注入的雜訊在 32K token 上維持在平坦高原，並在數百步內遺忘狀態脈衝，因為每次寫入都會沿當前 key 方向覆寫狀態；(iv) 逐 token 的量化成本會隨上下文被沖淡，而非累加。我們也修復了當由將模組融合為單一 GEMM 的核心來提供經逐模組校正的 NVFP4 checkpoint 時會出現的全局尺度不匹配，並證明經校準的 FP8 KV-cache 尺度不會造成效能損失。結果是：一套實用配方——量化所有項目，運送 KV 尺度——以及對為何混合 LLM 的循環半部是容易量化的半部的機制性解釋。Checkpoint: https://huggingface.co/minima-ai/mnma_qwen3.8_27b_nvfp4

English

Hybrid LLMs pair softmax attention with linear-attention layers such as Gated DeltaNet (GDN), whose recurrent state summarizes the context in fixed size. Early community 4-bit quantizations of Qwen3.8-27B (48 GDN layers, 16 attention layers) left the GDN block in 8- or 16-bit precision -- especially its decay and write-strength gates -- on the intuition that errors in a recurrence accumulate over long contexts. We test that intuition by building Minima: NVFP4 W4A4 on all 496 linear layers, GDN included. Across perplexity at 4K/32K, MMLU-Pro, GSM8K, AIME'25, GPQA-Diamond, LiveCodeBench, and RULER retrieval to 64K, Minima matches BF16 within seed noise (5-task average -0.52) while being the smallest (17.5 GiB) and fastest-prefill (+14-19%) recipe we compare, and its 32K perplexity gap shrinks with position. A four-part mechanism study explains why: (i) NVFP4's 16-element block scaling localizes the residual stream's extreme outliers, equalizing activation error across layer roles; (ii) the supposedly fragile gate projections are the least sensitive -- softplus/exponential and sigmoid parameterizations compress ~11% GEMM error to ~2% output error; (iii) the delta-rule recurrence holds injected noise at a flat plateau over 32K tokens and forgets a state impulse within hundreds of steps, because each write overwrites the state along the current key direction; (iv) the per-token quantization cost washes out with context instead of compounding. We also repair a global-scale mismatch that arises when per-module-calibrated NVFP4 checkpoints are served by kernels that fuse those modules into one GEMM, and show calibrated FP8 KV-cache scales are performance-free. The result: a practical recipe -- quantize everything, ship KV scales -- and a mechanistic account of why the recurrent half of a hybrid LLM is the easy half to quantize. Checkpoint: https://huggingface.co/minima-ai/mnma_qwen3.8_27b_nvfp4