H2O-Danube-1.8B 技術報告

H2O-Danube-1.8B Technical Report

January 30, 2024

作者: Philipp Singer, Pascal Pfeiffer, Yauhen Babakhin, Maximilian Jeblick, Nischay Dhankhar, Gabor Fodor, Sri Satish Ambati

cs.AI

摘要

我們介紹了H2O-Danube-1.8B，一個在1T tokens上訓練的1.8B語言模型，遵循LLama 2和Mistral的核心原則。我們利用並改進了各種技術來預訓練大型語言模型。儘管我們的模型在總token數方面比類似大小的參考模型訓練得少得多，但在眾多基準測試中展現出高競爭力的指標。我們另外釋出了一個經過監督微調和直接偏好優化訓練的聊天模型。我們通過Apache 2.0許可證公開提供H2O-Danube-1.8B，進一步使更廣泛的受眾在經濟上能夠使用LLM。

English

We present H2O-Danube-1.8B, a 1.8B language model trained on 1T tokens following the core principles of LLama 2 and Mistral. We leverage and refine various techniques for pre-training large language models. Although our model is trained on significantly fewer total tokens compared to reference models of similar size, it exhibits highly competitive metrics across a multitude of benchmarks. We additionally release a chat model trained with supervised fine-tuning followed by direct preference optimization. We make H2O-Danube-1.8B openly available under Apache 2.0 license further democratizing LLMs to a wider audience economically.