基于影响匹配的数据集蒸馏

摘要

我们从结果导向的角度重新审视了数据集蒸馏问题。与对齐过程替代指标（如逐步梯度或训练轨迹）不同，影响匹配（Inf-Match）方法对齐的是训练的最终结果：它学习一个紧凑的合成数据集，使其对收敛参数的影响与完整数据集的影响相匹配。具体而言，我们引入了一种完全可微的样本级影响估计器，能够量化添加或移除数据带来的参数变化，无需耗时的逆海森矩阵乘积或凸性假设。该估计器通过展开优化动态并应用一阶泰勒近似，以线性时间运行。随后，我们通过最小化合成数据集的影响与真实数据集影响之间的差异来学习合成数据集，从而实现结果对齐而非启发式的过程模仿。Inf-Match在标准分类基准上取得了最佳准确率。例如，在Tiny-ImageNet（IPC=10）上，Inf-Match达到了31.5%，相比NCFM提升了4.7%。除分类任务外，Inf-Match可扩展至Flickr30K上的视觉-语言蒸馏任务，其性能优于强大的过程匹配基线。例如，在使用200到1000个合成样本时，我们的方法在图像/文本检索任务上取得了领先的平均性能，比NCFM高出2.5%。代码将通过https://github.com/hrtan/infmatch发布。

English

We revisit dataset distillation from an outcome-centric perspective. Rather than aligning process surrogates (per-step gradients or training trajectories), Influence Matching (Inf-Match) aligns the final outcome of training: it learns a compact synthetic set whose effect on the converged parameters matches that of the full dataset. Concretely, we introduce a fully differentiable, sample-level influence estimator that quantifies parameter shifts from adding or removing data, without time-consuming inverse-Hessian products or convexity assumptions. The estimator runs in linear time by unrolling the optimization dynamics and applying a first-order Taylor approximation. We then learn the synthetic set by minimizing the mismatch between its influence and that of the real dataset, yielding outcome alignment rather than heuristic process imitation. Inf-Match delivers the best accuracy across standard classification benchmarks. For instance, on Tiny-ImageNet (IPC=10), Inf-Match attains 31.5\%, a +4.7\% improvement over NCFM. Beyond classification, Inf-Match scales to vision-language distillation on Flickr30K, outperforming strong process-matching baselines. For instance, with 200 to 1000 synthetic samples, our method achieved a leading impressive average on image/text retrieval tasks, higher than NCFM by 2.5\%. The code will be released via https://github.com/hrtan/infmatch.