反向训练来护理逆转诅咒

Reverse Training to Nurse the Reversal Curse

March 20, 2024

作者: Olga Golovneva, Zeyuan Allen-Zhu, Jason Weston, Sainbayar Sukhbaatar

cs.AI

摘要

大型语言模型（LLMs）存在一个令人惊讶的失败：当在“A具有特征B”上训练时，它们无法推广到“B是A的特征”，这被称为逆转诅咒。即使进行了数万亿标记的训练，由于Zipf定律，这个问题仍然会出现 - 因此即使我们在整个互联网上进行训练也是如此。本文提出了一种名为逆向训练的替代训练方案，其中所有单词都被使用两次，将可用标记数量翻倍。通过颠倒训练字符串并保留（即不颠倒）选择的子字符串（如实体），LLM在正向和逆向方向上进行训练。我们展示了数据匹配的逆向训练模型在标准任务上提供了比标准模型更优越的性能，并且计算匹配的逆向训练模型在逆转任务上提供了远超越的性能，有助于解决逆转诅咒问题。

English

Large language models (LLMs) have a surprising failure: when trained on "A has a feature B", they do not generalize to "B is a feature of A", which is termed the Reversal Curse. Even when training with trillions of tokens this issue still appears due to Zipf's law - hence even if we train on the entire internet. This work proposes an alternative training scheme, called reverse training, whereby all words are used twice, doubling the amount of available tokens. The LLM is trained in both forward and reverse directions by reversing the training strings while preserving (i.e., not reversing) chosen substrings, such as entities. We show that data-matched reverse-trained models provide superior performance to standard models on standard tasks, and compute-matched reverse-trained models provide far superior performance on reversal tasks, helping resolve the reversal curse issue.