ReFT：语言模型的表示微调

ReFT: Representation Finetuning for Language Models

April 4, 2024

作者: Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D. Manning, Christopher Potts

cs.AI

摘要

参数高效微调（PEFT）方法旨在通过对少量权重进行更新来调整大型模型。然而，许多先前的可解释性研究表明，表示编码了丰富的语义信息，这表明编辑表示可能是一种更强大的替代方法。在这里，我们通过开发一系列表示微调（ReFT）方法来探讨这一假设。ReFT方法在一个冻结的基础模型上运行，并学习对隐藏表示进行任务特定干预。我们定义了ReFT系列的一个强实例，即低秩线性子空间ReFT（LoReFT）。LoReFT可以直接替代现有的PEFT，并学习比先前最先进的PEFT高10倍至50倍的参数高效干预。我们在八个常识推理任务、四个算术推理任务、Alpaca-Eval v1.0和GLUE上展示了LoReFT。在所有这些评估中，LoReFT提供了效率和性能的最佳平衡，并几乎总是优于最先进的PEFT。我们在https://github.com/stanfordnlp/pyreft 上公开发布了一个通用的ReFT训练库。

English

Parameter-efficient fine-tuning (PEFT) methods seek to adapt large models via updates to a small number of weights. However, much prior interpretability work has shown that representations encode rich semantic information, suggesting that editing representations might be a more powerful alternative. Here, we pursue this hypothesis by developing a family of Representation Finetuning (ReFT) methods. ReFT methods operate on a frozen base model and learn task-specific interventions on hidden representations. We define a strong instance of the ReFT family, Low-rank Linear Subspace ReFT (LoReFT). LoReFT is a drop-in replacement for existing PEFTs and learns interventions that are 10x-50x more parameter-efficient than prior state-of-the-art PEFTs. We showcase LoReFT on eight commonsense reasoning tasks, four arithmetic reasoning tasks, Alpaca-Eval v1.0, and GLUE. In all these evaluations, LoReFT delivers the best balance of efficiency and performance, and almost always outperforms state-of-the-art PEFTs. We release a generic ReFT training library publicly at https://github.com/stanfordnlp/pyreft.