混合線性注意力大型語言模型中的巨量激活：注意力前尖峰與尖峰間平台

摘要

我們首次系統性地研究了層間交錯的混合線性注意力（HLA）大型語言模型中的大規模激活（MAs），並發現兩種與架構對齊的形態：MAs 會在全注意力層之前持續出現尖峰，形成注意力前尖峰（PAS），且可透過中間的線性注意力層持續存在，從而產生尖峰間平台（ISP）。隨著全注意力層變得更加密集，連續的 PAS 透過 ISP 日益相互連接，最終恢復全注意力 LLM 所具有的穩定 MA 形態。我們在五種線性注意力架構、六種混合配置、五個數據領域，以及參數總量從 1.2B 到 397B 的代表性開源混合模型中，確立了這種組織結構的重現性。基於 GDN 的混合模型在最高 1.3B 規模下的受控預訓練顯示，這兩種形態均在早期出現，並對輸出門控呈現不對稱反應：全注意力輸出門控會大幅衰減其絕對幅度，但不會消除其逐層組織結構；而移除 GDN 門控則僅造成相對溫和的放大。從機制上而言，我們的系統性離群值分析支持一個由 MA 抵消時機所主導的共享生命週期解釋。PAS 遵循局部化的寫入-下沉-抵消過程，而 ISP 的延長持續性則與延遲抵消一致。在全注意力極限下，此解釋恢復了全注意力 LLM 特有的穩定 MA 形態。我們的代碼可在 https://github.com/StartluxLabs/Massive-Activations-HLA 取得。

English

We present the first systematic study of Massive activations (MAs) in layer-interleaved HLA LLMs and uncover two architecture-aligned morphologies: MAs consistently spike immediately before full attention layers, forming pre-attention spikes (PAS), and can persist through intervening linear attention layers, giving rise to inter-spike plateaus (ISP). As full attention becomes denser, successive PAS become increasingly connected through ISP, ultimately recovering the stable MA morphology of full attention LLMs. We establish the recurrence of this organization across five linear attention architectures, six hybridization configurations, five data domains, and representative open-source hybrid models spanning 1.2B to 397B total parameters. Controlled pretraining of GDN-based hybrids at scales up to 1.3B shows that both morphologies emerge early and respond asymmetrically to output gating: full attention output gating strongly attenuates their absolute magnitudes without eliminating their layerwise organization, whereas removing GDN gates yields comparatively modest amplification. Mechanistically, our systematic-outlier analysis supports a shared lifecycle account governed by the timing of MA cancellation. PAS follows a localized write-sink-cancel process, while the extended persistence of ISP is consistent with delayed cancellation. At the full attention limit, this account recovers the stable MA morphology characteristic of full attention LLMs. Our code is available at https://github.com/StartluxLabs/Massive-Activations-HLA.