WARP：權重空間分析以恢復訓練資料組合

摘要

基礎模型通常會向公眾發布，但訓練它們所使用的數據配方——例如決定不同資料來源採樣方式的領域混合權重——卻鮮少公開。這造成了一種可及性不對稱：研究人員得以研究最終生成的模型，卻無法清楚了解產生這些模型的訓練資料分佈。過往推斷訓練資料的方法，例如成員推斷，僅能在個別樣本層級進行偵測，因此無法描述訓練語料庫的整體組成。我們提出了 WARP 框架，它能直接從微調模型釋出的權重中，還原其訓練混合比例。WARP 透過模型合併，在基礎模型與微調模型之間進行插值，生成近似缺失訓練軌跡的偽檢查點，並在權重空間中顯現出訓練資料的幾何足跡。從這些模擬足跡中，WARP 提取幾何特徵，並透過無參數的 softmax 讀出層或針對合成混合比例訓練的 MLP 投影器，將這些特徵映射到領域比例。在針對 BERT 和 GPT-2 進行的控制實驗中，WARP 恢復領域混合比例的平均絕對誤差（MAE）分別低至 0.046 和 0.104，優於成員推斷以及可存取真實訓練軌跡的變體方法。

English

Foundation models are routinely released to the public, yet the data recipes used to train them -- such as domain mixture weights that determine how different sources are sampled -- are rarely disclosed. This creates an access asymmetry: researchers study the resulting models but lack visibility into the training distribution that produces them. Prior works for inferring training data, such as membership inference, detect at the level of individual samples and thus cannot characterize the global composition of the training corpus. We introduce WARP, a framework that recovers a fine-tuned model's training mixtures directly from its released weights. WARP interpolates between the base and fine-tuned models using model merging, generating pseudo-checkpoints that approximate the missing training trajectory and expose a geometric footprint of the training data in the weight space. From these simulated footprints, WARP extracts geometric features and maps them to domain proportions using either a parameter-free softmax readout or an MLP projector trained on synthetic mixtures. In controlled experiments with BERT and GPT-2, WARP recovers domain mixtures with an average MAE as low as 0.046 and 0.104 respectively, outperforming membership inference and a variant with access to the true training trajectory.