模块化TTT：以可组合模块重新思考测试时训练

摘要

测试时训练（TTT）将序列建模视为一个在线学习问题，其中快速权重通过内部学习规则进行更新。尽管TTT变体的数量不断增加，现有方法通常将每种变体分别硬编码，这使得设计新的TTT方法以及隔离每个组件的作用变得困难。为解决这一问题，我们提出了模块化TTT（Modular TTT），该框架将内部学习者表示为有向无环图，并将快速权重网络、损失函数、学习率、权重衰减和归一化作为显式的设计维度。模块化TTT自动将基元级别的训练视图前向、训练视图反向和因果查询视图规则组合成完整的图级别TTT计算，包括快速权重状态转移。利用模块化TTT，我们系统地消融了TTT的各个组件，发现较小的学习率初始化、权重衰减和单层非线性能够提升性能，而MSE和内积损失表现相当。更深的快速权重网络和归一化往往损害性能，因为它们会导致过大的激活值，而残差连接和门控带来的可衡量收益甚微。基于这些发现，我们训练了最优变体模型，参数量为4.1亿和14.5亿，在1000亿token上进行训练，观察到训练损失和基准性能与Gated DeltaNet相当。

English

Test-time training (TTT) views sequence modeling as an online learning problem in which fast weights are updated by an internal learning rule. Despite the growing number of TTT variants, existing approaches typically hard-code each variant separately, which makes it difficult to design new TTT methods and to isolate the role of each component. To address this, we propose Modular TTT, a framework that represents the inner learner as a directed acyclic graph and exposes the fast-weight network, loss function, learning rate, weight decay, and normalization as explicit design dimensions. Modular TTT automatically composes primitive-level train-view forward, train-view backward, and causal query-view rules into the full graph-level TTT computation, including the fast-weight state transition. Using Modular TTT, we systematically ablate the components of TTT and find that small learning-rate initialization, weight decay, and a single-layer nonlinearity improve performance, while MSE and inner-product losses perform similarly. Deeper fast-weight networks and normalization tend to hurt performance because they induce excessively large activations, while residual connections and gating provide little measurable benefit. Guided by these findings, we train the best resulting variant as 410M- and 1.45B-parameter models on 100B tokens, and observe training loss and benchmark performance comparable to Gated DeltaNet.