xHC：擴展超連接

摘要

超連接（HC）將Transformer的殘差流擴展為N條並行流，提供一種超越模型寬度和深度的記憶體擴展方式。流形約束超連接（mHC）在規模上穩定了此公式。從N=1到N=4的巨大增益表明，殘差流擴展是一個有前景的擴展維度。然而，現有的HC系列方法通常在N=4止步。我們的實驗揭示了原因：將mHC擴展到此點之外會產生遞減的效能增益和急劇增加的訓練成本。我們將此限制歸因於兩個瓶頸：寫回資訊不足以應付不斷增長的串流數量，以及殘差混合生成的成本與N呈立方擴張。為了解決這兩個瓶頸，我們提出了xHC（擴展超連接），這是首個在N=4之外實現有意義擴展的HC系列方法。xHC結合了時序特徵增強以實現更豐富的寫回，以及稀疏殘差流架構，該架構僅更新N=16條流中的k=4條，同時保留對完整殘差狀態的密集存取。在18B和28B MoE模型上，xHC帶來了強大且一致的下游改進。在一個18B MoE模型上，與mHC相比，xHC將平均下游分數提升了4.0分，同時僅在原始基線之上增加了適度的訓練FLOPs。縮放定律實驗顯示，原始方法和mHC分別需要xHC的1.50倍和1.19倍計算量才能達到相同的損失。實際的大N訓練還需要控制來自擴展殘差狀態的記憶體流量。因此，我們引入了xHC-Flash，它將每子層的記憶體流量從73.5C降低到40C，與mHC在N=4時所需的34C相當，同時保留了完整xHC的增益。xHC和xHC-Flash共同使大N殘差流擴展在LLM預訓練中變得有效且實用。

English

Hyper-Connections (HC) expand the residual stream of Transformers into N parallel streams, providing a form of memory scaling beyond model width and depth. Manifold-Constrained HC (mHC) stabilizes this formulation at scale. The large gains from N{=}1 to N{=}4 suggest residual-stream expansion as a promising scaling axis. However, existing HC-family methods typically stop at N{=}4. Our experiments reveal why: scaling mHC beyond this point yields diminishing performance gains and rapidly increasing training cost. We attribute this limitation to two bottlenecks: insufficient write-back information for an expanding number of streams and residual-mixing generation whose cost scales cubically with N. To address both bottlenecks, we propose xHC (Expanded Hyper-Connections), the first HC-family method to achieve meaningful expansion beyond N{=}4. xHC combines temporal feature augmentation for richer write-back with a sparse residual-stream architecture that updates only k=4 of the N=16 streams while retaining dense access to the full residual state. Across 18B and 28B MoE models, xHC delivers strong and consistent downstream improvements. On an 18B MoE model, xHC improves the average downstream score by 4.0 points over mHC, while adding only modest training FLOPs over the vanilla baseline. Scaling-law experiments show that the vanilla and mHC require 1.50times and 1.19times the compute of xHC, respectively, to reach the same loss. Practical large-N training also requires controlling memory traffic from the expanded residual state. We therefore introduce xHC-Flash, which reduces the per-sublayer memory traffic from 73.5C to 40C, comparable to the 34C required by mHC at N{=}4, while retaining the gains of full xHC. Together, xHC and xHC-Flash make large-N residual-stream expansion effective and practical for LLM pre-training.

xHC：擴展超連接

xHC: Expanded Hyper-Connections

摘要

Support