FLM-101B：一個開放式LLM及如何在10萬美元預算下訓練它

FLM-101B: An Open LLM and How to Train It with $100K Budget

September 7, 2023

作者: Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Xuying Meng, Siqi Fan, Peng Han, Jing Li, Li Du, Bowen Qin, Zheng Zhang, Aixin Sun, Yequan Wang

cs.AI

摘要

大型語言模型（LLMs）在自然語言處理（NLP）和多模式任務中取得了顯著成功。儘管取得這些成就，它們的發展面臨兩個主要挑戰：（i）高計算成本；和（ii）難以進行公平客觀的評估。LLMs的成本極高，只有少數主要參與者才能進行培訓，這限制了研究和應用機會。這凸顯了成本效益高的LLM培訓的重要性。在本文中，我們利用增長策略顯著降低LLM培訓成本。我們展示了一個具有101B參數和0.31TB標記的LLM可以在10萬預算下進行培訓。我們還採用了一種系統評估範式，用於對LLMs進行智商評估，以補充現有更注重知識能力的評估。我們引入了我們的基準，其中包括對智能的重要方面進行評估，包括符號映射、規則理解、模式挖掘和抗干擾性。這些評估最大程度地減少了記憶的潛在影響。實驗結果表明，我們的模型FLM-101B，在10萬預算下進行培訓，與強大且知名的模型（例如GPT-3和GLM-130B）在IQ基準評估中取得了可比的表現，尤其是在訓練數據中未見過的情境下。FLM-101B的檢查點將在https://huggingface.co/CofeAI/FLM-101B上開源。

English

Large language models (LLMs) have achieved remarkable success in NLP and multimodal tasks. Despite these successes, their development faces two main challenges: (i) high computational cost; and (ii) difficulty in conducting fair and objective evaluations. LLMs are prohibitively expensive, making it feasible for only a few major players to undertake their training, thereby constraining both research and application opportunities. This underscores the importance of cost-effective LLM training. In this paper, we utilize a growth strategy to significantly reduce LLM training cost. We demonstrate that an LLM with 101B parameters and 0.31TB tokens can be trained on a 100K budget. We also adopt a systematic evaluation paradigm for the IQ evaluation of LLMs, in complement to existing evaluations that focus more on knowledge-oriented abilities. We introduce our benchmark including evaluations on important aspects of intelligence including symbolic mapping, itrule understanding, pattern mining, and anti-interference. Such evaluations minimize the potential impact of memorization. Experimental results show that our model FLM-101B, trained with a budget of 100K, achieves comparable performance to powerful and well-known models, eg GPT-3 and GLM-130B, especially in the IQ benchmark evaluations with contexts unseen in training data. The checkpoint of FLM-101B will be open-sourced at https://huggingface.co/CofeAI/FLM-101B.