教導語言模型通過互動示範自我改進

摘要

大型語言模型（LLMs）具有自我改進能力，透過提示它們分析並修訂自己的輸出，近期研究對此顯示出極大興趣。然而，較小模型顯示缺乏這種能力且難以學習，進一步擴大了最先進的LLMs與更具成本效益和更快速的模型之間的性能差距。為了縮小這一差距，我們引入了TriPosT，一種訓練算法，賦予較小模型這種自我改進能力，並證明我們的方法可以將LLaMA-7b在數學和推理任務上的表現提高最多達到7.13%。與先前的工作相比，我們通過使用較小模型與LLMs互動，收集反饋並改進自身生成，實現了這一點。然後，我們重播這種經驗以訓練小模型。我們對四個數學和推理數據集的實驗表明，從自身錯誤中學習並進行修正的互動經驗對於較小模型改善性能至關重要。

English

The self-improving ability of large language models (LLMs), enabled by prompting them to analyze and revise their own outputs, has garnered significant interest in recent research. However, this ability has been shown to be absent and difficult to learn for smaller models, thus widening the performance gap between state-of-the-art LLMs and more cost-effective and faster ones. To reduce this gap, we introduce TriPosT, a training algorithm that endows smaller models with such self-improvement ability, and show that our approach can improve a LLaMA-7b's performance on math and reasoning tasks by up to 7.13%. In contrast to prior work, we achieve this by using the smaller model to interact with LLMs to collect feedback and improvements on its own generations. We then replay this experience to train the small model. Our experiments on four math and reasoning datasets show that the interactive experience of learning from and correcting its own mistakes is crucial for small models to improve their performance.

教導語言模型通過互動示範自我改進

Teaching Language Models to Self-Improve through Interactive Demonstrations

摘要

Support