理解LLMs：從訓練到推論的全面概述

摘要

ChatGPT 的推出大幅提升了大型語言模型（LLMs）在應對下游任務中的利用率。在這一背景下，對於成本效益高的訓練和部署越來越受到關注。低成本的LLMs訓練和部署代表了未來的發展趨勢。本文回顧了大型語言模型訓練技術和推斷部署技術隨著這一新興趨勢的演變。訓練方面的討論包括多個方面，如數據預處理、訓練架構、預訓練任務、並行訓練，以及與模型微調相關的內容。在推斷方面，本文涵蓋了模型壓縮、並行計算、內存調度和結構優化等主題。同時探討了LLMs的應用並提供了對其未來發展的見解。

English

The introduction of ChatGPT has led to a significant increase in the utilization of Large Language Models (LLMs) for addressing downstream tasks. There's an increasing focus on cost-efficient training and deployment within this context. Low-cost training and deployment of LLMs represent the future development trend. This paper reviews the evolution of large language model training techniques and inference deployment technologies aligned with this emerging trend. The discussion on training includes various aspects, including data preprocessing, training architecture, pre-training tasks, parallel training, and relevant content related to model fine-tuning. On the inference side, the paper covers topics such as model compression, parallel computation, memory scheduling, and structural optimization. It also explores LLMs' utilization and provides insights into their future development.

理解LLMs：從訓練到推論的全面概述

Understanding LLMs: A Comprehensive Overview from Training to Inference

摘要

Support