混合线性注意力大语言模型中的巨量激活：注意力前尖峰与尖峰间平台期

摘要

我们首次系统研究了层交错式HLA大语言模型中的大规模激活（MAs），并揭示了两种与架构对齐的形态：MAs在全注意力层之前立即尖峰，形成注意力前尖峰（PAS），并且可以穿过中间的线性注意力层持续存在，从而产生尖峰间平台（ISP）。随着全注意力层变得更加密集，连续的PAS通过ISP日益连接起来，最终恢复全注意力大语言模型所特有的稳定MA形态。我们验证了这种组织在五种线性注意力架构、六种混合配置、五个数据领域以及参数总量从1.2B到397B的具有代表性的开源混合模型中的复现性。基于GDN的混合模型在高达1.3B参数规模下的受控预训练表明，这两种形态都在早期出现，并对输出门控表现出不对称响应：全注意力输出门控强烈削弱其绝对幅度，但不会消除其逐层组织；而移除GDN门控则产生相对适度的放大。在机制上，我们的系统性离群值分析支持一个由MA取消时机支配的共享生命周期解释。PAS遵循局部化的写入-吸收-取消过程，而ISP的延长持续与延迟取消一致。在全注意力极限下，这一解释恢复了全注意力大语言模型的特征性稳定MA形态。我们的代码可在https://github.com/StartluxLabs/Massive-Activations-HLA获取。

English

We present the first systematic study of Massive activations (MAs) in layer-interleaved HLA LLMs and uncover two architecture-aligned morphologies: MAs consistently spike immediately before full attention layers, forming pre-attention spikes (PAS), and can persist through intervening linear attention layers, giving rise to inter-spike plateaus (ISP). As full attention becomes denser, successive PAS become increasingly connected through ISP, ultimately recovering the stable MA morphology of full attention LLMs. We establish the recurrence of this organization across five linear attention architectures, six hybridization configurations, five data domains, and representative open-source hybrid models spanning 1.2B to 397B total parameters. Controlled pretraining of GDN-based hybrids at scales up to 1.3B shows that both morphologies emerge early and respond asymmetrically to output gating: full attention output gating strongly attenuates their absolute magnitudes without eliminating their layerwise organization, whereas removing GDN gates yields comparatively modest amplification. Mechanistically, our systematic-outlier analysis supports a shared lifecycle account governed by the timing of MA cancellation. PAS follows a localized write-sink-cancel process, while the extended persistence of ISP is consistent with delayed cancellation. At the full attention limit, this account recovers the stable MA morphology characteristic of full attention LLMs. Our code is available at https://github.com/StartluxLabs/Massive-Activations-HLA.