WARP: 恢复训练数据组合的权重空间分析

摘要

基础模型通常公开发布，但用于训练它们的数据配方（例如决定不同数据源采样方式的领域混合权重）却很少公开。这造成了访问不对称：研究人员可以研究训练得到的模型，却无法了解产生这些模型的训练数据分布。先前推断训练数据的方法（如成员推断）仅在单个样本层面进行检测，因此无法描述训练语料库的整体构成。我们提出WARP框架，该方法直接从微调模型发布后的权重中恢复其训练混合比例。WARP通过模型融合在基模型与微调模型之间进行插值，生成近似缺失训练轨迹的伪检查点，并在权重空间中揭示训练数据的几何特征足迹。基于这些模拟足迹，WARP提取几何特征，并通过免参数softmax读出器或在合成混合数据上训练的MLP投影器将其映射到领域比例。在BERT和GPT-2的受控实验中，WARP分别以平均绝对误差低至0.046和0.104恢复领域混合比例，性能优于成员推断方法，甚至优于能够获取真实训练轨迹的变体方法。

English

Foundation models are routinely released to the public, yet the data recipes used to train them -- such as domain mixture weights that determine how different sources are sampled -- are rarely disclosed. This creates an access asymmetry: researchers study the resulting models but lack visibility into the training distribution that produces them. Prior works for inferring training data, such as membership inference, detect at the level of individual samples and thus cannot characterize the global composition of the training corpus. We introduce WARP, a framework that recovers a fine-tuned model's training mixtures directly from its released weights. WARP interpolates between the base and fine-tuned models using model merging, generating pseudo-checkpoints that approximate the missing training trajectory and expose a geometric footprint of the training data in the weight space. From these simulated footprints, WARP extracts geometric features and maps them to domain proportions using either a parameter-free softmax readout or an MLP projector trained on synthetic mixtures. In controlled experiments with BERT and GPT-2, WARP recovers domain mixtures with an average MAE as low as 0.046 and 0.104 respectively, outperforming membership inference and a variant with access to the true training trajectory.