Orion-14B：開源多語言大型語言模型

Orion-14B: Open-source Multilingual Large Language Models

January 20, 2024

作者: Du Chen, Yi Huang, Xiaopu Li, Yongqiang Li, Yongqiang Liu, Haihui Pan, Leichao Xu, Dacheng Zhang, Zhipeng Zhang, Kun Han

cs.AI

摘要

在這項研究中，我們介紹了Orion-14B，這是一個具有140億參數的多語言大型語言模型集合。我們採用了數據排程方法，在來自英語、中文、日語、韓語和其他語言的文本中，訓練了一個基礎模型，該模型包含了2500億標記。此外，我們對一系列針對對話應用和其他特定用例進行了微調的模型。我們的評估結果表明，Orion-14B在廣泛的任務中實現了最先進的性能。我們將Orion-14B模型系列及其相關代碼公開提供，旨在激發未來在該領域的研究和實際應用。https://github.com/OrionStarAI/Orion

English

In this study, we introduce Orion-14B, a collection of multilingual large language models with 14 billion parameters. We utilize a data scheduling approach to train a foundational model on a diverse corpus of 2.5 trillion tokens, sourced from texts in English, Chinese, Japanese, Korean, and other languages. Additionally, we fine-tuned a series of models tailored for conversational applications and other specific use cases. Our evaluation results demonstrate that Orion-14B achieves state-of-the-art performance across a broad spectrum of tasks. We make the Orion-14B model family and its associated code publicly accessible https://github.com/OrionStarAI/Orion, aiming to inspire future research and practical applications in the field.