門控循環Transformer：透過循環調製實現的表達深度

摘要

擴展Transformer語言模型在表達力與記憶體效率之間造成了內在的張力。雖然各層獨特的權重保留了功能特化——從輸入錨定到抽象精煉——但這會產生可觀的記憶體佔用。相反地，標準的深度共享會強制執行均勻的轉換，消除表徵多樣性並降低建模品質。我們提出了門控遞迴Transformer（Gated Recurrent Transformer），一種遞迴深度Transformer架構，其中固定深度的前奏與尾聲區塊包夾著一個迭代R次的共享核心。受門控遞迴神經網路的啟發，我們採用輕量投影與逐元素更新閘——以隱藏狀態、固定前奏輸出及每一步重新採樣的雜訊為條件——來調制遞迴更新。這使得模型能夠在多次遞迴中將輸入特化到相同的少數幾層，而非需要許多獨特層來實現功能多樣性。在等FLOPS約束下，一個3層的門控遞迴Transformer在相似的訓練與推論FLOPS下，達到了12層GPT-2 Small基線的準確度，並在所有九個按預算縮放的單元中領先MoR與重尾深度採樣；在中型與大型規模下，它在標準token預算下接近密集模型的品質，且在中型規模下，一旦預算加倍，便能超越密集模型。在等參數量約束下，更深的遞迴在匹配的參數與資料預算下達到了2.76的驗證損失，而非遞迴對應模型為2.84。我們的結果表明，自適應深度重用是一種以參數換取品質的原則性策略：在大規模下，參數減少63%，峰值解碼記憶體減少59%，而編譯生成延遲僅增加10%。

English

Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layers preserve functional specialization---from input-grounding to abstract refinement---they incur a substantial memory footprint. Conversely, standard depth-sharing enforces uniform transformations that collapse representational diversity and degrade modeling quality. We introduce Gated Recurrent Transformer, a recurrent depth transformer where fixed-depth prelude and coda blocks bracket a single shared core iterated R times. Inspired by gated recurrent neural networks, we employ a lightweight projection and an elementwise update gate---conditioned on the hidden state, the fixed prelude output, and noise resampled at every step---to modulate the recurrent update. This allows the model to specialize the input to the same few layers across recurrences, rather than requiring many unique layers to achieve functional diversity. Under an isoFLOPS constraint, a 3-layer Gated Recurrent Transformer matches the accuracy of a 12-layer GPT-2 Small baseline with similar training and inference FLOPs, and leads MoR and heavy-tail depth sampling in all nine scale-by-budget cells; at medium and large scale it approaches dense quality at the standard token budget and overtakes it at medium scale once that budget is doubled. Under an isoPARAMS constraint, deeper recurrence achieves a 2.76 validation loss versus 2.84 for a non-recurrent counterpart at matched parameter and data budget. Our results demonstrate that adaptive depth reuse is a principled strategy for trading parameters for quality: at large scale, 63% fewer parameters and 59% less peak decoding memory for a 10% increase in compiled generation latency.