線性注意力架構：機制、權衡與跨層路由

摘要

自注意力机制使每个词元能够从整个上下文检索信息，但其在序列长度上的二次方成本限制了长上下文训练与推理。本文对softmax注意力与四种近期提出的循环线性注意力架构进行了比较研究：DeltaNet、门控DeltaNet、Kimi Delta注意力与门控DeltaNet-2。我们采用统一的循环记忆表示法对这些机制进行表述，明确揭示了它们在表达能力、记忆衰减、擦除与写入控制、训练吞吐量以及实现复杂度方面的差异。实验以15B词元训练的350M参数模型为核心，涵盖了优化器与学习率比较、混合堆叠与纯堆叠对比、序列长度运行时间测量、1.3B与3B参数规模下的更大DeltaNet运行实验，以及少量下游任务评估。文中报告的速度结果衡量的是训练吞吐量与迭代时间，未提供推理速度的经验基准。在报告的350M参数、15B词元调优范围内，采用Muon优化器的Kimi Delta注意力达到了最低的最终验证损失；使用AdamW训练的纯门控DeltaNet堆叠实现了最高归一化训练吞吐量；混合堆叠通常以吞吐量成本换取损失改善；而在我们评估的匹配架构设置中，Muon相对于AdamW始终能降低最终验证损失。我们引入并评估了面向DeltaNet风格记忆的轻量级跨层路由机制。最自然的基于DeltaNet的构想——将下层的delta规则写入误差转发至上层值目标——并未超越匹配基线。而将路由导向对齐的隐藏流并转发写入值则在所报告的匹配运行中带来了适度改进：跨层值路由（CLVR）同时降低了DeltaNet与门控DeltaNet的最终验证损失。

English

Self-attention lets each token retrieve information from the full context, but its quadratic cost in sequence length limits training and inference at long context. This paper presents a comparative study of softmax attention and four recent recurrent linear-attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2. We express these mechanisms in a common recurrent-memory notation, making explicit how they differ in expressivity, memory decay, erase and write control, training throughput, and implementation complexity. Our experiments center on 350M-parameter models trained for 15B tokens, and include optimizer and learning-rate comparisons, hybrid-versus-pure stack comparisons, sequence-length runtime measurements, larger DeltaNet runs at 1.3B and 3B parameters, and a small set of downstream evaluations. The reported speed results measure training throughput and iteration time; we do not provide an empirical inference-speed benchmark. Within the reported 350M-parameter, 15B-token sweep, Kimi Delta Attention with Muon reaches the lowest final validation loss, a pure Gated DeltaNet stack trained with AdamW has the highest normalized training throughput, hybrid stacks generally improve loss at a throughput cost, and Muon consistently lowers final validation loss relative to AdamW in the matched architecture settings we evaluate. We introduce and evaluate lightweight cross-layer routing mechanisms for DeltaNet-style memories. The most natural DeltaNet-inspired formulation, forwarding a lower layer's delta-rule write error into the next layer's value target, does not improve over matched baselines. Routing into the aligned hidden stream and forwarding the write value instead yields a modest improvement in the matched runs we report: Cross-Layer Value Routing (CLVR) lowers final validation loss for both DeltaNet and Gated DeltaNet.