H2O-Danube-1.8B 技术报告

摘要

我们介绍了H2O-Danube-1.8B，这是一个在1T标记上训练的1.8B语言模型，遵循了LLama 2和Mistral的核心原则。我们利用并改进了各种用于预训练大型语言模型的技术。尽管与类似规模的参考模型相比，我们的模型在训练的总标记数量上明显较少，但在多个基准测试中表现出高竞争力的指标。我们还发布了一个通过监督微调和直接偏好优化训练的聊天模型。我们以Apache 2.0许可证开放了H2O-Danube-1.8B，进一步使更广泛的受众经济上获得LLM的机会。

English

We present H2O-Danube-1.8B, a 1.8B language model trained on 1T tokens following the core principles of LLama 2 and Mistral. We leverage and refine various techniques for pre-training large language models. Although our model is trained on significantly fewer total tokens compared to reference models of similar size, it exhibits highly competitive metrics across a multitude of benchmarks. We additionally release a chat model trained with supervised fine-tuning followed by direct preference optimization. We make H2O-Danube-1.8B openly available under Apache 2.0 license further democratizing LLMs to a wider audience economically.

H2O-Danube-1.8B 技术报告

H2O-Danube-1.8B Technical Report

摘要

Support