解构注意力机制：探索高效语言建模的设计原则

摘要

Transformer语言模型的成功，普遍归功于其点积注意力机制，该机制融合了一系列关键设计原则：跨位置信息混合（实现多标记交互）、序列依赖的激活（注意力权重随输入自适应）、特定的数学形式（点积相似度加softmax加权），以及查询与键与动态隐藏状态的耦合（将注意力锚定在当前层）。然而，这些原则各自的必要性大多未经检验。在本研究中，我们通过设计可控变体，系统地解构了注意力机制，这些变体有选择性地放松了上述原则，既应用于所有层的统一架构，也应用于仅部分层保留标准注意力的混合架构。实证分析表明，标记混合机制不可或缺，其缺失会导致模型性能近乎随机，而精确的数学形式和序列依赖性则可大幅放宽，尤其是在仅部分层中保留时。令人惊讶的是，即便单独使用会失败的变体，在与标准注意力交替使用时也能展现出稳健的性能，突显了一种协同效应。这些发现深化了我们对注意力有效性真正基础的理解，并为在不牺牲性能的前提下简化语言模型开辟了新途径。

English

The success of Transformer language models is widely credited to their dot-product attention mechanism, which interweaves a set of key design principles: mixing information across positions (enabling multi-token interactions), sequence-dependent activations (where attention weights adapt to each input), a specific mathematical form (dot-product similarities plus softmax weighting), and coupling of queries and keys to evolving hidden states (grounding attention in the current layer). However, the necessity of each of these principles remains largely untested. In this work, we systematically deconstruct attention by designing controlled variants that selectively relax these principles, applied both uniformly across all layers and in hybrid architectures where only some layers retain standard attention. Our empirical analysis reveals that mechanisms for mixing tokens are indispensable, as their absence collapses models to near-random behavior, while the exact mathematical form and sequence dependency can be substantially relaxed, especially when preserved in just a subset of layers. Surprisingly, even variants that fail in isolation can achieve robust performance when interleaved with standard attention, highlighting a cooperative effect. These findings deepen our understanding of what truly underpins attention's effectiveness and open new avenues for simplifying language models without sacrificing performance.

解构注意力机制：探索高效语言建模的设计原则

Deconstructing Attention: Investigating Design Principles for Effective Language Modeling

摘要

Support