Delta激活：微调大型语言模型的一种表征方法

摘要

开源大型语言模型（LLMs）的成功，使得社区能够创建大量针对特定任务和领域进行后训练的模型。然而，由于元数据的不一致和存储库的非结构化，浏览和理解这些模型仍然具有挑战性。我们提出了Delta Activations方法，通过测量微调模型相对于基础模型内部激活的变化，将其表示为向量嵌入。这种表示方法能够有效地按领域和任务进行聚类，揭示模型生态中的结构。Delta Activations还展现出一些理想特性：它在不同的微调设置下表现出鲁棒性，并在微调数据集混合时展现出可加性。此外，我们展示了Delta Activations能够通过少样本微调嵌入任务，并进一步探索了其在模型选择和合并中的应用。我们希望Delta Activations能够促进公开可用模型的复用实践。代码可在https://github.com/OscarXZQ/delta_activations获取。

English

The success of powerful open source Large Language Models (LLMs) has enabled the community to create a vast collection of post-trained models adapted to specific tasks and domains. However, navigating and understanding these models remains challenging due to inconsistent metadata and unstructured repositories. We introduce Delta Activations, a method to represent finetuned models as vector embeddings by measuring shifts in their internal activations relative to a base model. This representation allows for effective clustering by domain and task, revealing structure in the model landscape. Delta Activations also demonstrate desirable properties: it is robust across finetuning settings and exhibits an additive property when finetuning datasets are mixed. In addition, we show that Delta Activations can embed tasks via few-shot finetuning, and further explore its use for model selection and merging. We hope Delta Activations can facilitate the practice of reusing publicly available models. Code is available at https://github.com/OscarXZQ/delta_activations.

Delta激活：微调大型语言模型的一种表征方法

Delta Activations: A Representation for Finetuned Large Language Models

摘要

Support