Delta激活：微调大型語言模型的一種表徵方式

摘要

強大的開源大型語言模型（LLMs）的成功，使得社群能夠創建大量針對特定任務和領域進行後訓練的模型。然而，由於不一致的元數據和非結構化的儲存庫，導航和理解這些模型仍然具有挑戰性。我們引入了Delta Activations方法，該方法通過測量微調模型相對於基礎模型內部激活的變化，將微調模型表示為向量嵌入。這種表示方式允許按領域和任務進行有效的聚類，從而揭示模型景觀中的結構。Delta Activations還展示了理想的特性：它在不同的微調設置下具有魯棒性，並且在微調數據集混合時表現出可加性。此外，我們展示了Delta Activations可以通過少樣本微調來嵌入任務，並進一步探索其在模型選擇和合併中的應用。我們希望Delta Activations能夠促進公開可用模型的重用實踐。代碼可在https://github.com/OscarXZQ/delta_activations獲取。

English

The success of powerful open source Large Language Models (LLMs) has enabled the community to create a vast collection of post-trained models adapted to specific tasks and domains. However, navigating and understanding these models remains challenging due to inconsistent metadata and unstructured repositories. We introduce Delta Activations, a method to represent finetuned models as vector embeddings by measuring shifts in their internal activations relative to a base model. This representation allows for effective clustering by domain and task, revealing structure in the model landscape. Delta Activations also demonstrate desirable properties: it is robust across finetuning settings and exhibits an additive property when finetuning datasets are mixed. In addition, we show that Delta Activations can embed tasks via few-shot finetuning, and further explore its use for model selection and merging. We hope Delta Activations can facilitate the practice of reusing publicly available models. Code is available at https://github.com/OscarXZQ/delta_activations.

Delta激活：微调大型語言模型的一種表徵方式

Delta Activations: A Representation for Finetuned Large Language Models

摘要

Support