DeepLoop：循环Transformer的深度缩放

摘要

循环Transformer通过应用紧凑的物理模块堆栈进行多轮计算来扩展顺序计算，增加展开深度而不增加存储参数。这种复用改变了残差缩放问题：在非捆绑Transformer中，每个残差分支接收并应用自身的参数更新；而在循环Transformer中，一个共享的更新聚合了来自重复访问的梯度，并由这些相同的访问在下一次线性化前向传播中重新读取。我们通过一个由访问对齐系数κ_R控制的一阶扰动界限形式化了这种捆绑深度效应。当访问解相关时，该界限恢复DeepNorm指数，但在保守的对齐状态下，随着循环次数在固定物理深度下增长，要求指数从1/4增加到1/2。由此产生的方法DeepLoop保留了Post-LN DeepNorm架构，并将展开深度N的α和β分别设为(2N)^{1/2}和(8N)^{-1/2}。在GPT-2 small和GPT-2 medium规模的GPT风格循环语言模型上，当没有物理模块被重新访问时，DeepLoop保持中性；一旦激活循环深度，它便改善了验证损失和下游准确性。这些结果表明，稳定的循环深度需要考虑参数访问次数而不仅仅是名义层数的残差缩放规则。

English

Looped Transformers scale sequential computation by applying a compact stack of physical blocks for multiple rounds, increasing unrolled depth without increasing stored parameters. This reuse changes the residual-scaling problem: in an untied Transformer, each residual branch receives and applies its own parameter update, whereas in a looped Transformer one shared update aggregates gradients from repeated visits and is read back by those same visits in the next linearized forward pass. We formalize this tied-depth effect through a first-order perturbation bound controlled by a visit-alignment coefficient κ_R. The bound recovers the DeepNorm exponent when visits decorrelate, but in the conservative aligned regime it requires the exponent to increase from 1/4 to 1/2 as loop count grows at fixed physical depth. The resulting method, DeepLoop, keeps the Post-LN DeepNorm architecture and sets α=(2N)^{1/2} and β=(8N)^{-1/2} for unrolled depth N. On GPT-style looped language models at GPT-2 small and GPT-2 medium scale, DeepLoop is neutral when no physical block is revisited and improves validation loss and downstream accuracy once recurrent depth is activated. These results show that stable recurrent depth requires residual scaling rules that account for parameter visits, not only nominal layer count.