在大语言模型预训练中缩放领域数据的重复使用

摘要

随着大语言模型规模不断扩大，其训练词元预算也必须相应增加，以维持合适的词元-参数比（TPP）。然而，高质量领域数据远比通用网络数据难以扩展。随着模型规模和训练词元预算的增长，高质量领域数据在训练数据混合中的占比往往会下降。重复使用现有的高质量数据是抵消这种稀释效应的有效方法，但过度重复可能导致过拟合。我们在实际的LLM扩展条件下研究这一权衡，其中训练词元预算随模型规模成比例增长。对于固定领域，我们首先发现，令人惊讶的是，在TPP固定的情况下，最优重复次数随模型规模的增大而略有增加。在不同领域之间，我们发现最优重复次数与该领域最终验证损失呈强负相关：验证损失越低的领域通常越能从更多重复中获益。相比之下，领域去重后的数据量与最优重复次数的相关性较弱。这些发现表明，在具有相同TPP的较小代理模型上调节得到的重复次数，可以为更大模型提供实用的估计。

English

As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(TPP\)). However, high-quality domain data is much harder to scale than general web data. As model size and the training-token budget increase, its fraction in the training mixture tends to decrease. Repeating the available high-quality data provides an effective way to counteract this dilution, but excessive repetition may lead to overfitting. We study this trade-off under practical LLM scaling, where the training-token budget grows proportionally with model size. For a fixed domain, we first find that, surprisingly at a fixed \(TPP\), the optimal repetition count mildly increases with model size. Across different domains, we find that the optimal repetition count is strongly negatively correlated with the final validation loss of a domain: domains with lower loss can generally benefit from more repetitions. In contrast, the amount of unique domain data is only weakly related to the optimal repetition count. These findings suggest that repetition counts tuned on smaller proxy models with the same \(TPP\) can provide a practical estimate for larger models.