多头专家混合模型

Multi-Head Mixture-of-Experts

April 23, 2024

作者: Xun Wu, Shaohan Huang, Wenhui Wang, Furu Wei

cs.AI

摘要

稀疏专家混合（SMoE）可以扩展模型容量，而不会显著增加训练和推理成本，但存在以下两个问题：（1）专家激活较低，仅有少数专家被激活以进行优化。（2）缺乏对单个标记内多个语义概念的细粒度分析能力。我们提出了多头专家混合（MH-MoE），它采用多头机制将每个标记分割为多个子标记。这些子标记然后被分配给并由一组不同的专家并行处理，然后无缝地重新集成到原始标记形式中。多头机制使模型能够共同关注来自不同专家的各种表示空间的信息，同时显著增强专家激活，从而加深上下文理解并减轻过拟合。此外，我们的MH-MoE易于实现，并与其他SMoE优化方法解耦，便于与其他SMoE模型集成以提高性能。在英语为中心的语言建模、多语言语言建模和遮蔽多模态建模任务上的大量实验结果表明了MH-MoE的有效性。

English

Sparse Mixtures of Experts (SMoE) scales model capacity without significant increases in training and inference costs, but exhibits the following two issues: (1) Low expert activation, where only a small subset of experts are activated for optimization. (2) Lacking fine-grained analytical capabilities for multiple semantic concepts within individual tokens. We propose Multi-Head Mixture-of-Experts (MH-MoE), which employs a multi-head mechanism to split each token into multiple sub-tokens. These sub-tokens are then assigned to and processed by a diverse set of experts in parallel, and seamlessly reintegrated into the original token form. The multi-head mechanism enables the model to collectively attend to information from various representation spaces within different experts, while significantly enhances expert activation, thus deepens context understanding and alleviate overfitting. Moreover, our MH-MoE is straightforward to implement and decouples from other SMoE optimization methods, making it easy to integrate with other SMoE models for enhanced performance. Extensive experimental results across three tasks: English-focused language modeling, Multi-lingual language modeling and Masked multi-modality modeling tasks, demonstrate the effectiveness of MH-MoE.