通过互动演示教授语言模型自我改进

摘要

近期的研究表明，通过促使大型语言模型（LLMs）分析和修订自身输出，这些模型具有自我改进的能力，引起了广泛关注。然而，这种能力被证明在较小的模型中缺失且难以学习，从而扩大了最先进的LLMs与更具成本效益和更快速模型之间的性能差距。为了缩小这一差距，我们提出了TriPosT，一种训练算法，赋予较小模型这种自我改进的能力，并展示了我们的方法可以将LLaMA-7b模型在数学和推理任务上的性能提高高达7.13%。与先前的工作相比，我们通过使用较小模型与LLMs进行交互，收集反馈和改进自身生成的方式来实现这一目标。然后，我们重放这一经验来训练小模型。我们在四个数学和推理数据集上的实验表明，从中互动学习并纠正自身错误的经验对于小模型改善性能至关重要。

English

The self-improving ability of large language models (LLMs), enabled by prompting them to analyze and revise their own outputs, has garnered significant interest in recent research. However, this ability has been shown to be absent and difficult to learn for smaller models, thus widening the performance gap between state-of-the-art LLMs and more cost-effective and faster ones. To reduce this gap, we introduce TriPosT, a training algorithm that endows smaller models with such self-improvement ability, and show that our approach can improve a LLaMA-7b's performance on math and reasoning tasks by up to 7.13%. In contrast to prior work, we achieve this by using the smaller model to interact with LLMs to collect feedback and improvements on its own generations. We then replay this experience to train the small model. Our experiments on four math and reasoning datasets show that the interactive experience of learning from and correcting its own mistakes is crucial for small models to improve their performance.

通过互动演示教授语言模型自我改进

Teaching Language Models to Self-Improve through Interactive Demonstrations

摘要

Support