基於影響匹配的數據集蒸餾

摘要

我們重新審視數據集蒸餾，從以結果為導向的觀點出發。與其對齊過程代理（每步梯度或訓練軌跡），影響匹配（Inf-Match）對齊訓練的最終結果：它學習一個緊湊的合成數據集，其對收斂參數的影響與完整數據集相匹配。具體而言，我們引入了一個完全可微分的、樣本層級的影響估計器，量化添加或移除數據所導致的參數變化，無需耗時的逆黑塞矩陣乘積或凸性假設。該估計器通過展開優化動力學並應用一階泰勒近似，以線性時間運行。然後，我們通過最小化合成數據集的影響與真實數據集影響之間的差異來學習合成數據集，從而實現結果對齊而非啟發式過程模仿。Inf-Match在標準分類基準上達到了最佳準確率。例如，在Tiny-ImageNet（IPC=10）上，Inf-Match達到31.5%，比NCFM提升了4.7%。超越分類任務，Inf-Match擴展到Flickr30K上的視覺-語言蒸餾，優於強大的過程匹配基線。例如，使用200到1000個合成樣本，我們的模型在圖像/文本檢索任務上取得了領先的出色平均表現，比NCFM高出2.5%。代碼將通過 https://github.com/hrtan/infmatch 發布。

English

We revisit dataset distillation from an outcome-centric perspective. Rather than aligning process surrogates (per-step gradients or training trajectories), Influence Matching (Inf-Match) aligns the final outcome of training: it learns a compact synthetic set whose effect on the converged parameters matches that of the full dataset. Concretely, we introduce a fully differentiable, sample-level influence estimator that quantifies parameter shifts from adding or removing data, without time-consuming inverse-Hessian products or convexity assumptions. The estimator runs in linear time by unrolling the optimization dynamics and applying a first-order Taylor approximation. We then learn the synthetic set by minimizing the mismatch between its influence and that of the real dataset, yielding outcome alignment rather than heuristic process imitation. Inf-Match delivers the best accuracy across standard classification benchmarks. For instance, on Tiny-ImageNet (IPC=10), Inf-Match attains 31.5\%, a +4.7\% improvement over NCFM. Beyond classification, Inf-Match scales to vision-language distillation on Flickr30K, outperforming strong process-matching baselines. For instance, with 200 to 1000 synthetic samples, our method achieved a leading impressive average on image/text retrieval tasks, higher than NCFM by 2.5\%. The code will be released via https://github.com/hrtan/infmatch.