LLM360：邁向完全透明的開源LLM模型

摘要

最近開源大型語言模型（LLM）的激增，如LLaMA、Falcon和Mistral，為人工智慧從業者和研究人員提供了多樣的選擇。然而，大多數LLM僅釋出部分產物，如最終模型權重或推論程式碼，而技術報告則越來越限制範圍，僅涉及高層次的設計選擇和表面統計。這些選擇阻礙了領域內的進展，降低了對LLM訓練透明度，迫使團隊重新發現訓練過程中的許多細節。我們提出LLM360，這是一個全面開源LLM的倡議，主張將所有訓練程式碼和數據、模型檢查點以及中間結果提供給社群使用。LLM360的目標是通過使端到端的LLM訓練過程對所有人透明且可重現，來支持開放和協作的人工智慧研究。作為LLM360的第一步，我們從頭開始預訓練了兩個7B參數的LLM，分別為Amber和CrystalCoder，包括它們的訓練程式碼、數據、中間檢查點和分析（網址：https://www.llm360.ai）。我們致力於通過這一開源努力不斷拓展LLM的界限。更大規模和更強大的模型正在進行中，並將在未來釋出。

English

The recent surge in open-source Large Language Models (LLMs), such as LLaMA, Falcon, and Mistral, provides diverse options for AI practitioners and researchers. However, most LLMs have only released partial artifacts, such as the final model weights or inference code, and technical reports increasingly limit their scope to high-level design choices and surface statistics. These choices hinder progress in the field by degrading transparency into the training of LLMs and forcing teams to rediscover many details in the training process. We present LLM360, an initiative to fully open-source LLMs, which advocates for all training code and data, model checkpoints, and intermediate results to be made available to the community. The goal of LLM360 is to support open and collaborative AI research by making the end-to-end LLM training process transparent and reproducible by everyone. As a first step of LLM360, we release two 7B parameter LLMs pre-trained from scratch, Amber and CrystalCoder, including their training code, data, intermediate checkpoints, and analyses (at https://www.llm360.ai). We are committed to continually pushing the boundaries of LLMs through this open-source effort. More large-scale and stronger models are underway and will be released in the future.

LLM360：邁向完全透明的開源LLM模型

LLM360: Towards Fully Transparent Open-Source LLMs

摘要

Support