快速權重注意力於持續學習

摘要

遞迴快速權重記憶與選擇性狀態空間模型將不斷擴展的上下文壓縮為固定大小的遞迴狀態，使狀態轉變成爲一種在線學習規則。我們在讀後寫自迴歸語義下研究此規則。對於此處考慮的前綴預測目標，步驟 \(t\) 揭露的局部快速記憶樣本是前綴對齊對 \((x_t, y_t) = (\phi(k_{t-1}), v_t)\)。常見的同步驟關聯 \((\phi(k_t), v_t)\) 保持因果性，但優化的是不同的內部目標。我們推導了平方誤差迴歸與負內積目標的歸一化一階更新。迴歸家族包含 Falcon-1（純量 NLMS 更新）、Falcon-2（其逐列擴展）以及 Falcon-3（滑動視窗小批量更新）；Falcon-1A/Falcon-2A/Falcon-3A 是對應的內積變體。我們提供了遞迴、遮罩平行與區塊平行形式，以及數值穩定的正衰減重整化。具代表性的變體在語言建模中保持競爭力，並改善了可變位數加法上的長度外推。此框架在遞迴序列模型中分離了時間對齊、可塑性、遺忘與有界複述。

English

Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step t is the prefix-aligned pair (x_t,y_t)=(ϕ(k_{t-1}),v_t). The common same-step association (ϕ(k_t),v_t) remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.