Puro-2B：Poor Lab 以不超過 5090 美元的預算，在 RTX 5090 上訓練的 Qwen2-1.5B

摘要

語言模型預訓練幾乎已成為高昂成本的代名詞，使學術界與開源社群中的大部分研究人員難以企及。儘管目前已存在强大的開源成果，包括開放權重模型與開源訓練配方，但一個兼具成本效益、硬體可及性與開源特性的預訓練配方長期以來仍然缺失。即便在小規模下，訓練 Llama-3.2-3B 的成本即超過 150 萬美元，而複現 SmolLM3-3B 也需要超過 70 萬美元。在本報告中，我們提出了一套旨在降低這道門檻的開放預訓練配方。利用此配方，我們在消費級 RTX 5090 GPU 上，以 FP8 精度從零開始訓練了一系列 Puro-2B 模型，涵蓋多達 1.4 兆個 token。該系列中的模型在 token 預算與所選配方變體上有所不同。我們的最佳模型以低於 6,900 美元的計算成本完成訓練，在我們的評估協議下性能接近 Qwen2.5-1.5B。這種成本效率得益於多種方法的結合，包括硬體選擇、低精度訓練、超球優化、課程模型平均以及資料配方。除了配方本身之外，我們還提供了兩項額外的成果。首先，我們在整個 Puro-2B 系列上推導出「Puro 成本擴展定律」，用以關聯訓練成本與模型平均性能；擬合結果顯示，約 4,400 美元（低於 5,090 美元）的成本即足以達到 Qwen2-1.5B 的性能。其次，作為一項端到端案例研究，我們考察了預訓練資料課程如何影響後訓練之後的下游性能。此類受控研究之所以可行，是因為我們能夠取得完整的預訓練管線，而非僅有模型權重。我們在 Apache 2.0 許可下，於 https://huggingface.co/collections/thu-pacman/puro-2b 發布了 Puro-2B 的完整訓練配方，包括資料、程式碼與模型權重。

English

Language model pretraining has become almost synonymous with prohibitive cost, placing it out of reach for much of the academic and open-source communities. Although strong open-source efforts already exist, including open-weight models and open-source training recipes, a cost-efficient, hardware-accessible, and open-source pretraining recipe has long been missing. Even at a small scale, training Llama-3.2-3B costs over \1.5M, and reproducing SmolLM3-3B needs over 700K. In this report, we present an open pretraining recipe designed to lower this barrier. Using this recipe, we train a collection of Puro-2B models from scratch on up to 1.4 trillion tokens with FP8 precision on consumer-grade RTX 5090 GPUs. The models in the collection differ in token budgets and selected recipe variants. Our best model is trained at a compute cost of less than \6.9K and approaches Qwen2.5-1.5B performance under our evaluation protocol. This cost efficiency is enabled by a combination of approaches, including hardware selection, low-precision training, hyperball optimization, curriculum model averaging, and the data recipe. Beyond the recipe itself, we provide two additional results. First, across the Puro-2B collection, we derive a Puro Cost Scaling Law that relates training cost to average model performance; the fitted law suggests that about 4.4K, less than \$5,090, is sufficient to reach the performance of Qwen2-1.5B. Second, as an end-to-end case study, we examine how pretraining data curricula shape downstream performance after post-training. Such controlled studies are enabled by having access to the full pretraining pipeline rather than model weights alone. We release the full training recipe for Puro-2B, including data, code, and model weights under Apache 2.0 at https://huggingface.co/collections/thu-pacman/puro-2b.