Puro-2B：贫困实验室在RTX 5090上以5090美元预算训练的Qwen2-1.5B

摘要

语言模型预训练几乎已成为高昂成本的代名词，使得学术界和开源社区的大部分群体难以企及。尽管目前已有强大的开源成果，包括开放权重模型和开源训练配方，但一种成本高效、硬件可及且开源的预训练配方长期以来一直缺失。即便在小规模下，训练Llama-3.2-3B的成本也超过150万美元，而复现SmolLM3-3B需要超过70万美元。在本报告中，我们提出了一种旨在降低这一门槛的开放预训练配方。利用该配方，我们以FP8精度在消费级RTX 5090 GPU上从头训练了一系列Puro-2B模型，词元规模最高达1.4万亿。该系列中的模型在词元预算和所选配方变体上有所不同。我们最佳模型的计算成本低于6900美元，在我们的评估协议下接近Qwen2.5-1.5B的性能。这种成本效率得益于多种方法的结合，包括硬件选择、低精度训练、超球优化、课程模型平均以及数据配方。除配方本身外，我们还提供了两项额外成果。首先，基于Puro-2B系列，我们推导出Puro成本缩放定律，将训练成本与平均模型性能相关联；拟合结果表明，约4400美元（低于5090美元）即可达到Qwen2-1.5B的性能水平。其次，作为一项端到端案例研究，我们考察了预训练数据课程如何塑造后训练之后的下游性能。此类受控研究之所以成为可能，是因为我们能够访问完整的预训练流水线，而非仅获得模型权重。我们在https://huggingface.co/collections/thu-pacman/puro-2b 以Apache 2.0许可发布了Puro-2B的完整训练配方，包括数据、代码和模型权重。

English

Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over \1.5M, and reproducing SmolLM3-3B needs over 700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs. The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than \6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about 4.4K, less than \$5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B, including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b.