Maglev：滑動循環記憶

摘要

我們提出，一種具有固定大小記憶體的循環Transformer架構，它推廣了滑動視窗注意力，同時在訓練期間保持可並行化。由兩個耦合模型組成：一個預填充器 Q，它利用完整注意力（在實務上，我們對 Q 使用交錯的全注意力和滑動視窗注意力，因為這能帶來更強的效能。其基本要求是 Q 必須比 P 更具表達能力，並能存取完整的歷史記錄）來產生記憶目標 m'_t；以及一個解碼器 P，它僅使用滑動視窗注意力和循環 K/V 注入，來產生用於下一個詞元預測的解碼器記憶 m_t。我們以記憶一致性損失來訓練，使 m_t 與 m'_t 對齊，從而在推論時可僅使用 P。實驗上，相比滑動視窗和潛在循環Transformer基線，改善了驗證損失以及下游預訓練基準。此外，在 P 與 Q 之間共享參數可減少參數記憶體，同時保留大部分的增益。

English

We introduce , a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. consists of two coupled models: a prefiller Q, which leverages full attentionIn practice, we use interleaved full and sliding-window attention for Q, as this yields stronger performance. The essential requirement is that Q be more expressive than P, with access to the full history. to produce memory targets m'_t, and a decoder P, which uses only sliding-window attention and recurrent K/V injection to produce decoder memories m_t for next-token prediction. We train with a memory consistency loss that aligns m_t with m'_t, allowing inference to use P alone. Empirically, improves validation loss and downstream pretraining benchmarks over sliding-window and latent recurrent transformer baselines. Moreover, sharing parameters between P and Q reduces parameter memory while preserving most of the gains.