狀態-預測分離假說

摘要

Transformer 使用相同的前向計算流來同時預測下一個 token 並儲存對未來 token 預測有用的狀態。我們提出了「狀態-預測分離假說」：將這兩個角色分離，能帶來更佳的語言建模表現。我們設計了一種 Transformer 變體，採用兩個計算流來區分這兩項功能，並在不同規模下進行預訓練實驗。實驗結果顯示，狀態-預測分離能一致地提升資料與計算效率，改善驗證損失，並在下游任務上平均優於標準 Transformer 2 到 3 個百分點。我們亦進行了廣泛的實證分析，排除了潛在的混雜因子，並證明我們設計所引入的梯度具有根本性的差異。

English

Transformers use the same forward computation stream to both predict the next token and store useful state for future token predictions. We formulate the state-prediction separation hypothesis: disentangling the two roles yields better language modeling performance. We design a Transformer variant that uses two computation streams to separate the two functions, and conduct pretraining experiments across various scales. Our experiments show that state-prediction separation consistently offers better data and compute efficiencies, improving validation loss and outperforming standard Transformers by 2--3 percentage points on average on downstream tasks. We also conduct extensive empirical analysis that rules out potential confounders and demonstrates the fundamental difference in the gradients our design entails.