揭秘數據受限語言模型預訓練中的訓練時增強

摘要

隨著AI實驗室接近數據上限，計算能力已超越高品質新文本生成的速率，語言模型預訓練正轉向數據受限、計算充裕的階段，這需要在固定語料庫上進行高效的多輪次訓練。標準自回歸（AR）預訓練在這種情況下會嚴重過擬合，在達到最佳點後便持續惡化。我們研究將訓練階段的數據增強作為正則化手段，以減輕此過擬合問題，並能在同一數據上進行數百輪次的有效訓練。我們提出三種正交的AR預訓練增強類別：詞元級噪聲（遮蔽、隨機替換）、序列排列（從右到左預測、填空預測），以及目標偏移預測（對於i > 1，預測x_{t+i}）。通過系統性消融實驗，我們發現個別增強方法能延遲過擬合並降低相對於基線的驗證損失，其中隨機詞元替換在個別方法中達到最佳最低損失。結合不同增強類別則能進一步降低最低驗證損失。我們的實驗證明，數據增強能緩解AR預訓練的數據效率低下問題，並為數據受限的階段提供有前景的解決方案。\footnote{所有代碼與數據均可於 https://github.com/michaelchen-lab/data-augmentations-for-pretraining 取得。}

English

As AI labs approach a data ceiling where compute capacity outpaces the rate of new high-quality text generation, language model pretraining is shifting toward a data-constrained, compute-abundant regime that demands productive multi-epoch training on fixed corpora. Standard autoregressive (AR) pretraining overfits severely in this setting, reaching its optimum early and then continuously deteriorating. We investigate training-time data augmentation as a regularizer to mitigate this overfitting and enable productive training for hundreds of epochs on the same data. We introduce three orthogonal categories of augmentation for AR pretraining: token-level noise (masking, random replacement), sequence permutations (right-to-left prediction, Fill-in-the-Middle), and target offset prediction (x_{t+i} for i > 1). Through systematic ablations, we find that individual augmentations delay overfitting and lower validation loss relative to the baseline, with random token replacement achieving the best minimum loss among individual methods. Combining augmentation categories further lowers the minimum validation loss. Our experiments demonstrate that data augmentations mitigate AR pretraining's data inefficiency and offer a promising solution to the data-constrained regime~\footnote{All code and data are available at https://github.com/ michaelchen-lab/ data-augmentations-for-pretraining.