Maglev: 滑动循环记忆

摘要

我们提出了一种循环Transformer架构，其具有固定大小的记忆，能够泛化滑动窗口注意力，同时在训练期间保持可并行化。该架构由两个耦合模型组成：一个预填充器Q，利用完整注意力（在实际应用中，我们对Q采用交错的全注意力和滑动窗口注意力，因为这样能带来更强的性能；基本要求是Q比P更具表达力，并能访问完整历史）来生成记忆目标m'_t；以及一个解码器P，仅使用滑动窗口注意力和循环K/V注入来生成用于下一个词元预测的解码器记忆m_t。我们使用记忆一致性损失来训练该架构，使m_t与m'_t对齐，从而允许推理时仅使用P。实验结果表明，与滑动窗口和潜在循环Transformer基线相比，该方法在验证损失和下游预训练基准上均有改进。此外，在P和Q之间共享参数可在保留大部分收益的同时减少参数存储。

English

We introduce , a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. consists of two coupled models: a prefiller Q, which leverages full attentionIn practice, we use interleaved full and sliding-window attention for Q, as this yields stronger performance. The essential requirement is that Q be more expressive than P, with access to the full history. to produce memory targets m'_t, and a decoder P, which uses only sliding-window attention and recurrent K/V injection to produce decoder memories m_t for next-token prediction. We train with a memory consistency loss that aligns m_t with m'_t, allowing inference to use P alone. Empirically, improves validation loss and downstream pretraining benchmarks over sliding-window and latent recurrent transformer baselines. Moreover, sharing parameters between P and Q reduces parameter memory while preserving most of the gains.