更好地与指令来回翻译对齐

Better Alignment with Instruction Back-and-Forth Translation

August 8, 2024

作者: Thao Nguyen, Jeffrey Li, Sewoong Oh, Ludwig Schmidt, Jason Weston, Luke Zettlemoyer, Xian Li

cs.AI

摘要

我们提出了一种新方法，即指令来回翻译，用于构建基于世界知识的高质量合成数据，以对齐大型语言模型（LLMs）。给定来自网络语料库的文档，我们利用Li等人（2023a）提出的反向翻译方法生成和策划合成指令，并根据初始文档进一步改写响应以提高其质量。利用生成的（反向翻译指令，改写响应）对进行微调，在AlpacaEval上获得比使用其他常见指令数据集（如Humpback、ShareGPT、Open Orca、Alpaca-GPT4和Self-instruct）更高的胜率。我们还证明，利用LLM改写响应优于直接蒸馏，而两个生成的文本分布在嵌入空间中表现出明显的差异。进一步分析表明，我们的反向翻译指令比其他来源的合成指令质量更高，而我们的响应比蒸馏获得的响应更加多样化和复杂。总体而言，我们发现指令来回翻译结合了网络上发现的信息多样性和数量，同时确保了对齐所必需的响应质量，融合了两全其美。

English

We propose a new method, instruction back-and-forth translation, to construct high-quality synthetic data grounded in world knowledge for aligning large language models (LLMs). Given documents from a web corpus, we generate and curate synthetic instructions using the backtranslation approach proposed by Li et al.(2023a), and rewrite the responses to improve their quality further based on the initial documents. Fine-tuning with the resulting (backtranslated instruction, rewritten response) pairs yields higher win rates on AlpacaEval than using other common instruction datasets such as Humpback, ShareGPT, Open Orca, Alpaca-GPT4 and Self-instruct. We also demonstrate that rewriting the responses with an LLM outperforms direct distillation, and the two generated text distributions exhibit significant distinction in embedding space. Further analysis shows that our backtranslated instructions are of higher quality than other sources of synthetic instructions, while our responses are more diverse and complex than those obtained from distillation. Overall we find that instruction back-and-forth translation combines the best of both worlds -- making use of the information diversity and quantity found on the web, while ensuring the quality of the responses which is necessary for effective alignment.