DELIFT：資料高效語言模型指令微調

DELIFT: Data Efficient Language model Instruction Fine Tuning

November 7, 2024

作者: Ishika Agarwal, Krishna Killamsetty, Lucian Popa, Marina Danilevksy

cs.AI

摘要

對大型語言模型（LLMs）進行微調對於提升其在特定任務上的表現至關重要，但由於存在冗餘或無信息的數據，這往往需要耗費大量資源。為解決這種低效問題，我們引入了DELIFT（Data Efficient Language model Instruction Fine-Tuning），這是一種新穎的算法，系統地優化了微調的三個關鍵階段中的數據選擇：（1）指令微調，（2）特定任務的微調（例如推理、問答），以及（3）持續微調（例如整合新數據版本）。與現有方法不同，這些方法著重於單階段優化或依賴於計算密集型的梯度計算，DELIFT在所有階段都能高效運作。我們方法的核心是一種成對效用度量標準，該標準量化了一個數據樣本對於改善模型對其他樣本的響應有多有益，有效地測量了信息價值相對於模型當前能力的情況。通過利用應用於這個度量標準的不同子模模函數，DELIFT選擇出多樣化和最佳子集，這些子集在微調的所有階段都是有用的。通過在各種任務和模型規模上進行的實驗表明，DELIFT可以將微調數據大小減少多達70％，同時不影響性能，提供了顯著的計算節省，並在效率和功效方面優於現有方法。

English

Fine-tuning large language models (LLMs) is essential for enhancing their performance on specific tasks but is often resource-intensive due to redundant or uninformative data. To address this inefficiency, we introduce DELIFT (Data Efficient Language model Instruction Fine-Tuning), a novel algorithm that systematically optimizes data selection across the three key stages of fine-tuning: (1) instruction tuning, (2) task-specific fine-tuning (e.g., reasoning, question-answering), and (3) continual fine-tuning (e.g., incorporating new data versions). Unlike existing methods that focus on single-stage optimization or rely on computationally intensive gradient calculations, DELIFT operates efficiently across all stages. Central to our approach is a pairwise utility metric that quantifies how beneficial a data sample is for improving the model's responses to other samples, effectively measuring the informational value relative to the model's current capabilities. By leveraging different submodular functions applied to this metric, DELIFT selects diverse and optimal subsets that are useful across all stages of fine-tuning. Experiments across various tasks and model scales demonstrate that DELIFT can reduce the fine-tuning data size by up to 70% without compromising performance, offering significant computational savings and outperforming existing methods in both efficiency and efficacy.