HybridNorm：透過混合歸一化實現穩定且高效的Transformer訓練

HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization

March 6, 2025

作者: Zhijian Zhuo, Yutao Zeng, Ya Wang, Sijun Zhang, Jian Yang, Xiaoqing Li, Xun Zhou, Jinwen Ma

cs.AI

摘要

Transformer已成為廣泛機器學習任務，特別是在大型語言模型（LLMs）中的實際架構標準。儘管其表現卓越，但在訓練深度Transformer網絡時仍存在挑戰，尤其是關於層歸一化的位置。雖然Pre-Norm結構因其更顯著的恆等路徑而便於訓練，但其性能往往不如Post-Norm。本文提出了一種簡單而有效的混合歸一化策略——HybridNorm，它結合了Pre-Norm和Post-Norm的優勢。具體而言，HybridNorm在注意力機制中採用QKV歸一化，並在每個Transformer塊的前饋網絡（FFN）中使用Post-Norm。這種設計不僅穩定了訓練，還提升了性能，尤其是在LLMs的背景下。在密集和稀疏架構中的全面實驗表明，HybridNorm始終優於Pre-Norm和Post-Norm方法，在各種基準測試中達到了最先進的結果。這些發現凸顯了HybridNorm作為一種更穩定、更有效的技術，在改進深度Transformer模型的訓練和性能方面的潛力。代碼將公開提供，詳見https://github.com/BryceZhuo/HybridNorm。

English

Transformers have become the de facto architecture for a wide range of machine learning tasks, particularly in large language models (LLMs). Despite their remarkable performance, challenges remain in training deep transformer networks, especially regarding the location of layer normalization. While Pre-Norm structures facilitate easier training due to their more prominent identity path, they often yield suboptimal performance compared to Post-Norm. In this paper, we propose HybridNorm, a straightforward yet effective hybrid normalization strategy that integrates the advantages of both Pre-Norm and Post-Norm approaches. Specifically, HybridNorm employs QKV normalization within the attention mechanism and Post-Norm in the feed-forward network (FFN) of each transformer block. This design not only stabilizes training but also enhances performance, particularly in the context of LLMs. Comprehensive experiments in both dense and sparse architectures show that HybridNorm consistently outperforms both Pre-Norm and Post-Norm approaches, achieving state-of-the-art results across various benchmarks. These findings highlight the potential of HybridNorm as a more stable and effective technique for improving the training and performance of deep transformer models. %Code will be made publicly available. Code is available at https://github.com/BryceZhuo/HybridNorm.