TiKMiX：將數據影響納入語言模型預訓練的動態混合中

TiKMiX: Take Data Influence into Dynamic Mixture for Language Model Pre-training

August 25, 2025

作者: Yifan Wang, Binbin Liu, Fengze Liu, Yuanfan Guo, Jiyao Deng, Xuecheng Wu, Weidong Zhou, Xiaohuan Zhou, Taifeng Wang

cs.AI

摘要

語言模型預訓練所使用的數據混合策略，是其最終性能的基石。然而，靜態的混合策略並非最優，因為模型在訓練過程中對不同數據領域的學習偏好會動態變化。關鍵在於，如何以計算高效的方式觀察這些不斷演變的偏好，仍是一大挑戰。為此，我們提出了TiKMiX，一種根據模型演變偏好動態調整數據混合的方法。TiKMiX引入了群體影響力（Group Influence），這是一種評估數據領域對模型影響的高效指標。該指標使得數據混合問題能夠被表述為尋找一個最優、影響力最大化的分佈。我們通過兩種方法來解決這一問題：TiKMiX-D用於直接優化，而TiKMiX-M則利用回歸模型預測更優的混合比例。我們訓練了不同參數量的模型，處理了高達1萬億個token的數據。TiKMiX-D在僅使用20%計算資源的情況下，超越了REGMIX等最先進方法的性能。TiKMiX-M在9個下游基準測試中平均帶來了2%的性能提升。我們的實驗表明，模型的數據偏好隨訓練進度和規模而演變，並且我們證明，基於群體影響力（這些偏好的直接衡量指標）動態調整數據混合，能夠顯著提升性能，緩解靜態比例下數據消化不足的問題。

English

The data mixture used in the pre-training of a language model is a cornerstone of its final performance. However, a static mixing strategy is suboptimal, as the model's learning preferences for various data domains shift dynamically throughout training. Crucially, observing these evolving preferences in a computationally efficient manner remains a significant challenge. To address this, we propose TiKMiX, a method that dynamically adjusts the data mixture according to the model's evolving preferences. TiKMiX introduces Group Influence, an efficient metric for evaluating the impact of data domains on the model. This metric enables the formulation of the data mixing problem as a search for an optimal, influence-maximizing distribution. We solve this via two approaches: TiKMiX-D for direct optimization, and TiKMiX-M, which uses a regression model to predict a superior mixture. We trained models with different numbers of parameters, on up to 1 trillion tokens. TiKMiX-D exceeds the performance of state-of-the-art methods like REGMIX while using just 20% of the computational resources. TiKMiX-M leads to an average performance gain of 2% across 9 downstream benchmarks. Our experiments reveal that a model's data preferences evolve with training progress and scale, and we demonstrate that dynamically adjusting the data mixture based on Group Influence, a direct measure of these preferences, significantly improves performance by mitigating the underdigestion of data seen with static ratios.