探索指令调优的格式一致性

摘要

指导调整已经成为增强大型语言模型以遵循人类指令的一种有前途的方法。研究表明，在训练数据中增加指令的多样性和数量可以持续增强泛化性能，这有助于最近的一项努力，即收集各种指令并将现有的指导调整数据集整合到更大的集合中。然而，不同用户有其独特表达指令的方式，不同数据集之间指令风格和格式存在变化，即格式不一致。在这项工作中，我们研究了格式不一致如何影响指导调整的性能。我们提出了一个名为“统一指导调整”（UIT）的框架，该框架调用OpenAI API在不同的指导调整数据集之间进行自动格式转换。我们展示了UIT成功提高了对未见指令的泛化性能，突显了格式一致性对指导调整的重要性。为了使UIT框架更实用，我们进一步提出了一种基于困惑度的去噪方法，以减少自动格式转换的噪声。我们还训练了一个较小的离线模型，其具有与OpenAI API相当的格式转换能力，以在实践中降低成本。

English

Instruction tuning has emerged as a promising approach to enhancing large language models in following human instructions. It is shown that increasing the diversity and number of instructions in the training data can consistently enhance generalization performance, which facilitates a recent endeavor to collect various instructions and integrate existing instruction tuning datasets into larger collections. However, different users have their unique ways of expressing instructions, and there often exist variations across different datasets in the instruction styles and formats, i.e., format inconsistency. In this work, we study how format inconsistency may impact the performance of instruction tuning. We propose a framework called "Unified Instruction Tuning" (UIT), which calls OpenAI APIs for automatic format transfer among different instruction tuning datasets. We show that UIT successfully improves the generalization performance on unseen instructions, which highlights the importance of format consistency for instruction tuning. To make the UIT framework more practical, we further propose a novel perplexity-based denoising method to reduce the noise of automatic format transfer. We also train a smaller offline model that achieves comparable format transfer capability than OpenAI APIs to reduce costs in practice.

探索指令调优的格式一致性

Exploring Format Consistency for Instruction Tuning

摘要

Support