锥形语言模型

摘要

現代語言模型（包括Transformer、遞迴及記憶變體）共享同一套基礎架構：由一組相同層級堆疊而成，其中參數均勻分配於各層深度。此設計源自原始Transformer的預設，且至今大致未變。然而，越來越多的證據顯示，各層對最終輸出的貢獻並非均等——後期層主要是在精煉殘差流，而非轉換它。我們因此探討：參數容量是否應反映這種不對稱性。在控制實驗中，我們發現，在固定預算下，將更多容量分配給早期層、較少分配給後期層，能改善困惑度，勝過均勻寬度的基準線；而反向分配則會損害性能。基於此結果，我們提出「錐形語言模型」（Tapered Language Models, TLMs）的架構原則：在固定總預算下，讓含參數的組件在深度方向上呈現單調遞減的錐形。MLP是此實現的自然場域：它在所有現代LM家族中佔據參數多數，並以寬度作為單一、清晰的變異軸。在三個模型規模與四種架構（Transformer、Gated Attention、Hope-attention 與 Titans）上，透過平滑餘弦排程對MLP寬度進行錐形化，無論在困惑度還是下游基準測試表現上，均一致優於均勻寬度的基準線，且無需額外參數或計算成本。這些發現確立了「深度感知容量分配」作為一種簡單、與架構無關的語言模型設計軸——一個隱藏在眼前、無需成本的槓桿。

English

Modern language models, including transformer, recurrent, and memory-based variants, share a common chassis: a stack of identical layers in which parameters are allocated uniformly across depth. This is a default inherited from the original transformer and largely unchanged since, yet a growing body of evidence suggests that layers contribute non-uniformly to the final output, with later layers refining the residual stream rather than transforming it. We ask whether parameter capacity should reflect this asymmetry. Our controlled experiment shows that, under a fixed budget, allocating more capacity to earlier layers and less to later layers improves perplexity over a uniform-width baseline, while the reverse allocation hurts. Building on this result, we introduce Tapered Language Models (TLMs), an architectural principle in which a parameter-bearing component is monotonically tapered across depth under a fixed total budget. MLPs are the natural site for this instantiation: they dominate parameter count across all modern LM families and expose width as a single, clean axis of variation. Across three model scales and four architectures (Transformer, Gated Attention, Hope-attention, and Titans), tapering MLP width via a smooth cosine schedule consistently improves perplexity and downstream benchmark performance over uniform baselines, at no additional parameter or compute cost. These findings establish depth-aware capacity allocation as a simple, architecture-agnostic axis of language model design, a free lever hidden in plain sight.