面向持续学习的快速权重注意力

摘要

递归快速权重记忆与选择性状态空间模型将不断扩展的上下文压缩为固定大小的递归状态，使状态转换成为一种在线学习规则。我们在写后读自回归语义下研究该规则。对于本文考虑的前缀预测目标，第t步揭示的局部快速记忆示例是前缀对齐对(x_t,y_t)=(ϕ(k_{t-1}),v_t)。常见的同步关联(ϕ(k_t),v_t)仍保持因果性，但优化的是不同的内部目标。我们推导了平方误差回归和负内积目标的归一化一阶更新。回归族包括Falcon-1（标量NLMS更新）、Falcon-2（其逐列扩展）和Falcon-3（滑动窗口小批量更新）；Falcon-1A/Falcon-2A/Falcon-3A是对应的内积变体。我们提供了递归、掩码并行和分块并行三种形式，以及数值稳定的正衰减重归一化方法。代表性变体在语言建模中保持竞争力，并在可变位数加法上改进了长度外推。该框架将递归序列模型中的时间对齐、可塑性、遗忘和有界复述明确区分开来。

English

Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step t is the prefix-aligned pair (x_t,y_t)=(ϕ(k_{t-1}),v_t). The common same-step association (ϕ(k_t),v_t) remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.