状态-预测分离假说

摘要

Transformer使用相同的前向计算流同时完成两件事：预测下一个token以及为未来的token预测存储有用状态。我们提出了状态-预测分离假说：将这两个角色解耦能够带来更好的语言建模性能。我们设计了一种基于双计算流的Transformer变体，以分离这两个功能，并在不同规模上进行了预训练实验。实验结果表明，状态-预测分离能够持续提升数据与计算效率，降低验证损失，并在下游任务上平均优于标准Transformer 2至3个百分点。我们还开展了广泛的实证分析，排除了潜在的混淆因素，并证明了我们设计所引入的梯度本质差异。

English

Transformers use the same forward computation stream to both predict the next token and store useful state for future token predictions. We formulate the state-prediction separation hypothesis: disentangling the two roles yields better language modeling performance. We design a Transformer variant that uses two computation streams to separate the two functions, and conduct pretraining experiments across various scales. Our experiments show that state-prediction separation consistently offers better data and compute efficiencies, improving validation loss and outperforming standard Transformers by 2--3 percentage points on average on downstream tasks. We also conduct extensive empirical analysis that rules out potential confounders and demonstrates the fundamental difference in the gradients our design entails.