xHC：扩展的超连接

摘要

超连接（Hyper-Connections, HC）将Transformer的残差流扩展为N个并行流，提供了一种超越模型宽度和深度的记忆扩展形式。流形约束超连接（Manifold-Constrained HC, mHC）在大规模下稳定了该结构。从N=1到N=4的显著增益表明，残差流扩展是一条有前景的扩展维度。然而，现有HC家族方法通常止步于N=4。我们的实验揭示了原因：将mHC扩展到更大规模会导致性能增益递减且训练成本急剧增加。我们将这一限制归因于两个瓶颈：一是扩展的流数导致回写信息不足，二是残差混合生成的复杂度随N呈三次方增长。为解决这两个瓶颈，我们提出xHC（扩展超连接），这是首个能在N>4时实现有意义扩展的HC家族方法。xHC融合了时间特征增强以提供更丰富的回写信息，以及一种稀疏残差流架构——该架构仅更新N=16个流中的k=4个，同时保留对完整残差状态的密集访问权限。在18B和28B MoE模型上，xHC带来了强劲且一致的下游性能提升。在18B MoE模型上，xHC相比mHC将平均下游得分提升了4.0分，同时相对于原始基线仅增加了适度的训练FLOPs。扩展定律实验表明，为达到相同损失，原始模型和mHC所需计算量分别是xHC的1.50倍和1.19倍。大规模N的实际训练还需控制扩展残差状态带来的内存流量。为此我们引入xHC-Flash，该方法将每子层内存流量从73.5C降至40C，与mHC在N=4时所需的34C相当，同时保留了完整xHC的性能增益。xHC与xHC-Flash共同使大规模N残差流扩展在大语言模型预训练中变得高效且实用。

English

Hyper-Connections (HC) expand the residual stream of Transformers into N parallel streams, providing a form of memory scaling beyond model width and depth. Manifold-Constrained HC (mHC) stabilizes this formulation at scale. The large gains from N{=}1 to N{=}4 suggest residual-stream expansion as a promising scaling axis. However, existing HC-family methods typically stop at N{=}4. Our experiments reveal why: scaling mHC beyond this point yields diminishing performance gains and rapidly increasing training cost. We attribute this limitation to two bottlenecks: insufficient write-back information for an expanding number of streams and residual-mixing generation whose cost scales cubically with N. To address both bottlenecks, we propose xHC (Expanded Hyper-Connections), the first HC-family method to achieve meaningful expansion beyond N{=}4. xHC combines temporal feature augmentation for richer write-back with a sparse residual-stream architecture that updates only k=4 of the N=16 streams while retaining dense access to the full residual state. Across 18B and 28B MoE models, xHC delivers strong and consistent downstream improvements. On an 18B MoE model, xHC improves the average downstream score by 4.0 points over mHC, while adding only modest training FLOPs over the vanilla baseline. Scaling-law experiments show that the vanilla and mHC require 1.50times and 1.19times the compute of xHC, respectively, to reach the same loss. Practical large-N training also requires controlling memory traffic from the expanded residual state. We therefore introduce xHC-Flash, which reduces the per-sublayer memory traffic from 73.5C to 40C, comparable to the 34C required by mHC at N{=}4, while retaining the gains of full xHC. Together, xHC and xHC-Flash make large-N residual-stream expansion effective and practical for LLM pre-training.

xHC：扩展的超连接

xHC: Expanded Hyper-Connections

摘要

Support