ICon：自動化數據選擇中的上下文貢獻

摘要

指令微調的數據選擇對於提升大型語言模型（LLMs）的性能和降低訓練成本至關重要。然而，現有的自動化選擇方法要么依賴於計算成本高昂的基於梯度的度量，要么依賴於人工設計的啟發式方法，這些方法可能無法充分利用數據的內在屬性。本文提出了一種新穎的無梯度方法——基於上下文學習的貢獻度量（ICon），該方法利用上下文學習（ICL）的隱式微調特性來衡量樣本貢獻，無需梯度計算或人工指標工程。ICon提供了一種計算效率高的替代方案，相較於基於梯度的方法，並減少了基於啟發式方法中固有的人為歸納偏差。ICon由三個組件構成，通過評估在ICL隱式學習下的性能變化來識別高貢獻數據。在三個LLMs上進行的廣泛實驗，涵蓋12個基準測試和5個配對評估集，證明了ICon的有效性。值得注意的是，在LLaMA3.1-8B上，使用15% ICon選擇數據訓練的模型比使用完整數據集訓練的模型性能高出5.42個百分點，並且比廣泛使用的選擇方法的最佳性能高出2.06個百分點。我們進一步分析了ICon選擇的高貢獻樣本，這些樣本展示了多樣化的任務和適宜的難度水平，而不僅僅是最難的樣本。

English

Data selection for instruction tuning is essential for improving the performance of Large Language Models (LLMs) and reducing training cost. However, existing automated selection methods either depend on computationally expensive gradient-based measures or manually designed heuristics, which may fail to fully exploit the intrinsic attributes of data. In this paper, we propose In-context Learning for Contribution Measurement (ICon), a novel gradient-free method that takes advantage of the implicit fine-tuning nature of in-context learning (ICL) to measure sample contribution without gradient computation or manual indicators engineering. ICon offers a computationally efficient alternative to gradient-based methods and reduces human inductive bias inherent in heuristic-based approaches. ICon comprises three components and identifies high-contribution data by assessing performance shifts under implicit learning through ICL. Extensive experiments on three LLMs across 12 benchmarks and 5 pairwise evaluation sets demonstrate the effectiveness of ICon. Remarkably, on LLaMA3.1-8B, models trained on 15% of ICon-selected data outperform full datasets by 5.42% points and exceed the best performance of widely used selection methods by 2.06% points. We further analyze high-contribution samples selected by ICon, which show both diverse tasks and appropriate difficulty levels, rather than just the hardest ones.

ICon：自動化數據選擇中的上下文貢獻

ICon: In-Context Contribution for Automatic Data Selection

摘要

Summary

Support

Support