模組化TTT：將測試時訓練重新思考為可組合模組

摘要

測試時訓練（TTT）將序列建模視為一個線上學習問題，其中快速權重透過內部學習規則進行更新。儘管 TTT 變體的數量持續增加，現有方法通常分別對每個變體進行硬編碼，這使得設計新的 TTT 方法以及區分每個組成部分的作用變得困難。為了解決這個問題，我們提出模組化 TTT（Modular TTT），此框架將內部學習器表示為有向無環圖，並將快速權重網路、損失函數、學習率、權重衰減與正規化暴露為明確的設計維度。模組化 TTT 自動將基元層級的訓練視角前向、訓練視角反向與因果查詢視角規則組合成完整的圖層級 TTT 計算，包括快速權重狀態轉換。我們使用模組化 TTT 系統性地消融 TTT 的各個組成部分，發現較小的學習率初始化、權重衰減與單層非線性有助於提升效能，而 MSE 與內積損失的表現則相近。較深的快速權重網路與正規化往往會損害效能，因為它們會導致過大的激活值；而殘差連接與門控則幾乎沒有提供可衡量的益處。根據這些發現，我們將效能最佳的變體訓練為 4.1 億與 14.5 億參數的模型（使用 1000 億個 token），並觀察到其訓練損失與基準評測表現與 Gated DeltaNet 相當。

English

Test-time training (TTT) views sequence modeling as an online learning problem in which fast weights are updated by an internal learning rule. Despite the growing number of TTT variants, existing approaches typically hard-code each variant separately, which makes it difficult to design new TTT methods and to isolate the role of each component. To address this, we propose Modular TTT, a framework that represents the inner learner as a directed acyclic graph and exposes the fast-weight network, loss function, learning rate, weight decay, and normalization as explicit design dimensions. Modular TTT automatically composes primitive-level train-view forward, train-view backward, and causal query-view rules into the full graph-level TTT computation, including the fast-weight state transition. Using Modular TTT, we systematically ablate the components of TTT and find that small learning-rate initialization, weight decay, and a single-layer nonlinearity improve performance, while MSE and inner-product losses perform similarly. Deeper fast-weight networks and normalization tend to hurt performance because they induce excessively large activations, while residual connections and gating provide little measurable benefit. Guided by these findings, we train the best resulting variant as 410M- and 1.45B-parameter models on 100B tokens, and observe training loss and benchmark performance comparable to Gated DeltaNet.