门控循环Transformer：循环调制带来的表达深度

摘要

扩展Transformer语言模型在表达力与内存效率之间存在着固有的张力。虽然各层独立权重保留了功能特化——从输入锚定到抽象精化——但会带来可观的内存占用。相反，标准深度共享强制施加统一的变换，压缩了表征多样性并降低了建模质量。我们提出门控循环Transformer（Gated Recurrent Transformer），一种循环深度Transformer架构，其中固定深度的前奏块和尾声块围绕一个被迭代R次的共享核心。受门控循环神经网络启发，我们采用轻量级投影和逐元素更新门——以隐状态、固定前奏输出以及每步重采样的噪声为条件——来调节循环更新。这使得模型能够在多次循环中针对同一少数层特化输入，而无需大量独立层即可实现功能多样性。在等FLOPS约束下，3层门控循环Transformer在训练和推理FLOPs相近的情况下达到了12层GPT-2 Small基线的准确率，并在全部九个按预算划分的规模单元中领先MoR和重尾深度采样方法；在标准token预算下，中等及大规模下它接近稠密模型质量，一旦预算翻倍，在中等规模下超越稠密模型。在等参数约束下，更深的循环在匹配参数和数据预算下达到2.76的验证损失，而非循环对应模型为2.84。我们的结果表明，自适应深度复用是一种以参数换质量的系统性策略：在大规模下，参数减少63%，峰值解码内存减少59%，而编译后生成延迟仅增加10%。

English

Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layers preserve functional specialization---from input-grounding to abstract refinement---they incur a substantial memory footprint. Conversely, standard depth-sharing enforces uniform transformations that collapse representational diversity and degrade modeling quality. We introduce Gated Recurrent Transformer, a recurrent depth transformer where fixed-depth prelude and coda blocks bracket a single shared core iterated R times. Inspired by gated recurrent neural networks, we employ a lightweight projection and an elementwise update gate---conditioned on the hidden state, the fixed prelude output, and noise resampled at every step---to modulate the recurrent update. This allows the model to specialize the input to the same few layers across recurrences, rather than requiring many unique layers to achieve functional diversity. Under an isoFLOPS constraint, a 3-layer Gated Recurrent Transformer matches the accuracy of a 12-layer GPT-2 Small baseline with similar training and inference FLOPs, and leads MoR and heavy-tail depth sampling in all nine scale-by-budget cells; at medium and large scale it approaches dense quality at the standard token budget and overtakes it at medium scale once that budget is doubled. Under an isoPARAMS constraint, deeper recurrence achieves a 2.76 validation loss versus 2.84 for a non-recurrent counterpart at matched parameter and data budget. Our results demonstrate that adaptive depth reuse is a principled strategy for trading parameters for quality: at large scale, 63% fewer parameters and 59% less peak decoding memory for a 10% increase in compiled generation latency.