无需调整指令的指令跟随

Instruction Following without Instruction Tuning

September 21, 2024

作者: John Hewitt, Nelson F. Liu, Percy Liang, Christopher D. Manning

cs.AI

摘要

指导调整通常意味着在指导-响应对上微调语言模型。我们发现两种适应（调整）形式相对于指导调整来说存在不足，但仍会产生指导遵循；我们将其称为隐式指导调整。我们首先发现指导-响应对并非必要：仅在响应上进行训练，没有任何相应的指导，也会导致指导遵循。这表明预训练模型具有一个指导-响应映射，通过教授模型所需的响应分布来揭示。然而，我们随后发现并不需要教授所需的响应分布：在像诗歌这样的狭窄领域数据上进行指导-响应训练仍会导致像食谱生成这样的广泛指导遵循行为。特别是，当指导与微调领域中的指导非常不同时，模型的响应不会遵循微调领域的风格。为了开始解释隐式指导调整，我们假设对语言模型的分布进行非常简单的更改会导致指导遵循。我们通过手写基于规则的语言模型来支持这一点，在与预训练模型的专家乘积中实现指导遵循。这些规则是逐渐增加结束序列的概率，惩罚重复，并均匀改变15个单词的概率。总之，未经设计以产生指导遵循的调整可以隐式地实现这一目标。

English

Instruction tuning commonly means finetuning a language model on instruction-response pairs. We discover two forms of adaptation (tuning) that are deficient compared to instruction tuning, yet still yield instruction following; we call this implicit instruction tuning. We first find that instruction-response pairs are not necessary: training solely on responses, without any corresponding instructions, yields instruction following. This suggests pretrained models have an instruction-response mapping which is revealed by teaching the model the desired distribution of responses. However, we then find it's not necessary to teach the desired distribution of responses: instruction-response training on narrow-domain data like poetry still leads to broad instruction-following behavior like recipe generation. In particular, when instructions are very different from those in the narrow finetuning domain, models' responses do not adhere to the style of the finetuning domain. To begin to explain implicit instruction tuning, we hypothesize that very simple changes to a language model's distribution yield instruction following. We support this by hand-writing a rule-based language model which yields instruction following in a product-of-experts with a pretrained model. The rules are to slowly increase the probability of ending the sequence, penalize repetition, and uniformly change 15 words' probabilities. In summary, adaptations made without being designed to yield instruction following can do so implicitly.