精細混合專家的規模定律

摘要

專家混合（Mixture of Experts，MoE）模型已成為降低大型語言模型計算成本的主要解決方案。在這項工作中，我們分析了它們的擴展變數範圍，並探討其擴展性質。具體來說，我們引入了一個新的超參數，即粒度（granularity），通過調整它，可以精確控制專家的大小。基於此，我們建立了細粒度MoE的擴展規律，考慮了訓練標記數量、模型大小和粒度。利用這些規律，我們為給定計算預算推導出最佳的訓練配置。我們的研究結果不僅顯示MoE模型始終優於密集Transformer，還凸顯了在擴大模型大小和訓練預算的情況下，密集和MoE模型之間的效率差距擴大。此外，我們證明，在幾乎任何計算預算下，將MoE中專家的大小設置為與前向傳播層相同的常見做法並不是最佳選擇。

English

Mixture of Experts (MoE) models have emerged as a primary solution for reducing the computational cost of Large Language Models. In this work, we analyze their scaling properties, incorporating an expanded range of variables. Specifically, we introduce a new hyperparameter, granularity, whose adjustment enables precise control over the size of the experts. Building on this, we establish scaling laws for fine-grained MoE, taking into account the number of training tokens, model size, and granularity. Leveraging these laws, we derive the optimal training configuration for a given computational budget. Our findings not only show that MoE models consistently outperform dense Transformers but also highlight that the efficiency gap between dense and MoE models widens as we scale up the model size and training budget. Furthermore, we demonstrate that the common practice of setting the size of experts in MoE to mirror the feed-forward layer is not optimal at almost any computational budget.

精細混合專家的規模定律

Scaling Laws for Fine-Grained Mixture of Experts

摘要

Support