大型語言模型預訓練中的領域數據重複擴展

摘要

隨著大型語言模型規模的擴展，其訓練token預算也必須相應增加，以維持適當的每參數token數比例（\(TPP\)）。然而，高品質的領域資料遠比一般網路資料更難以擴展。隨著模型規模與訓練token預算的增加，高品質領域資料在訓練混合中的占比往往會下降。重複使用現有的高品質資料提供了抵消這種稀釋效應的有效途徑，但過度的重複可能導致過擬合。我們在實際的大型語言模型擴展情境下研究此權衡取捨，其中訓練token預算隨模型規模成比例增長。針對固定領域，我們首先發現，令人驚訝的是，在固定的\(TPP\)下，最適重複次數會隨模型規模略微增加。跨不同領域比較時，我們發現最適重複次數與該領域的最終驗證損失呈強烈負相關：損失較低的領域通常能從更多的重複中受益。相比之下，獨特領域資料的數量與最適重複次數僅有微弱的關聯。這些發現表明，在具有相同\(TPP\)的較小代理模型上調整出的重複次數，可為較大模型提供實用的估計依據。

English

As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(TPP\)). However, high-quality domain data is much harder to scale than general web data. As model size and the training-token budget increase, its fraction in the training mixture tends to decrease. Repeating the available high-quality data provides an effective way to counteract this dilution, but excessive repetition may lead to overfitting. We study this trade-off under practical LLM scaling, where the training-token budget grows proportionally with model size. For a fixed domain, we first find that, surprisingly at a fixed \(TPP\), the optimal repetition count mildly increases with model size. Across different domains, we find that the optimal repetition count is strongly negatively correlated with the final validation loss of a domain: domains with lower loss can generally benefit from more repetitions. In contrast, the amount of unique domain data is only weakly related to the optimal repetition count. These findings suggest that repetition counts tuned on smaller proxy models with the same \(TPP\) can provide a practical estimate for larger models.