KromHC：基于克罗内克积残差矩阵的流形约束超连接

KromHC: Manifold-Constrained Hyper-Connections with Kronecker-Product Residual Matrices

January 29, 2026

作者: Wuyang Zhou, Yuxuan Gu, Giorgos Iacovides, Danilo Mandic

cs.AI

摘要

超连接（HC）在神经网络中的成功也凸显了其训练不稳定性和可扩展性受限的问题。流形约束超连接（mHC）通过将残差连接空间投影到Birkhoff多胞体上来缓解这些挑战，但仍面临两个问题：1）其迭代式Sinkhorn-Knopp（SK）算法并不总能产生精确的双随机残差矩阵；2）mHC的参数复杂度高达难以承受的O(n³C)，其中n为残差流宽度，C为特征维度。近期提出的mHC-lite通过Birkhoff-von-Neumann定理对残差矩阵进行重参数化以保证双随机性，但其参数复杂度也面临阶乘级爆炸问题，达到O(nC·n!)。为解决这两大挑战，我们提出KromHC方法，通过小型双随机矩阵的Kronecker积来参数化mHC中的残差矩阵。通过沿张量化残差流的每个模态对因子残差矩阵实施流形约束，KromHC在保证残差矩阵精确双随机性的同时，将参数复杂度降至O(n²C)。综合实验表明，KromHC在显著减少可训练参数的同时，达到甚至超越了当前最先进的mHC变体性能。代码已开源：https://github.com/wz1119/KromHC。

English

The success of Hyper-Connections (HC) in neural networks (NN) has also highlighted issues related to its training instability and restricted scalability. The Manifold-Constrained Hyper-Connections (mHC) mitigate these challenges by projecting the residual connection space onto a Birkhoff polytope, however, it faces two issues: 1) its iterative Sinkhorn-Knopp (SK) algorithm does not always yield exact doubly stochastic residual matrices; 2) mHC incurs a prohibitive O(n^3C) parameter complexity with n as the width of the residual stream and C as the feature dimension. The recently proposed mHC-lite reparametrizes the residual matrix via the Birkhoff-von-Neumann theorem to guarantee double stochasticity, but also faces a factorial explosion in its parameter complexity, O left( nC cdot n! right). To address both challenges, we propose KromHC, which uses the Kronecker products of smaller doubly stochastic matrices to parametrize the residual matrix in mHC. By enforcing manifold constraints across the factor residual matrices along each mode of the tensorized residual stream, KromHC guarantees exact double stochasticity of the residual matrices while reducing parameter complexity to O(n^2C). Comprehensive experiments demonstrate that KromHC matches or even outperforms state-of-the-art (SOTA) mHC variants, while requiring significantly fewer trainable parameters. The code is available at https://github.com/wz1119/KromHC.