反向缩放：当更大不一定更好时

Inverse Scaling: When Bigger Isn't Better

June 15, 2023

作者: Ian R. McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, Andrew Gritsevskiy, Daniel Wurgaft, Derik Kauffman, Gabriel Recchia, Jiacheng Liu, Joe Cavanagh, Max Weiss, Sicong Huang, The Floating Droid, Tom Tseng, Tomasz Korbak, Xudong Shen, Yuhui Zhang, Zhengping Zhou, Najoung Kim, Samuel R. Bowman, Ethan Perez

cs.AI

摘要

研究规模定律发现，大型语言模型（LMs）在规模增加（模型大小、训练数据和计算资源）的情况下，整体损失呈可预测的改善趋势。在这里，我们提出了一项主张的证据，即LMs可能表现出逆向缩放，或者随着规模增加，任务表现更差，例如由于训练目标和数据中存在的缺陷。我们通过对通过公开比赛收集的11个数据集进行的实证研究，即“逆向缩放奖”，展示了逆向缩放的证据，该比赛设有丰厚的奖金池。通过对数据集的分析，以及文献中发现的其他示例，我们确定了逆向缩放的四个潜在原因：（i）更倾向于重复记忆序列而不是遵循上下文指令，（ii）模仿训练数据中的不良模式，（iii）任务包含一个LMs可能专注于的简单干扰任务，而不是更困难的真实任务，以及（iv）关于任务的正确但具有误导性的少样本演示。我们将获奖数据集发布在https://inversescaling.com/data，以便进一步研究逆向缩放。我们的任务有助于发现U形和倒U形缩放趋势，其中初始趋势发生逆转，表明规模趋势在预测更大规模模型行为方面的可靠性不如先前所理解的那样。总的来说，我们的结果表明，增加模型规模本身可能不会带来进展的任务存在，并且对于训练语言模型的数据和目标需要更加慎重地考虑。

English

Work on scaling laws has found that large language models (LMs) show predictable improvements to overall loss with increased scale (model size, training data, and compute). Here, we present evidence for the claim that LMs may show inverse scaling, or worse task performance with increased scale, e.g., due to flaws in the training objective and data. We present empirical evidence of inverse scaling on 11 datasets collected by running a public contest, the Inverse Scaling Prize, with a substantial prize pool. Through analysis of the datasets, along with other examples found in the literature, we identify four potential causes of inverse scaling: (i) preference to repeat memorized sequences over following in-context instructions, (ii) imitation of undesirable patterns in the training data, (iii) tasks containing an easy distractor task which LMs could focus on, rather than the harder real task, and (iv) correct but misleading few-shot demonstrations of the task. We release the winning datasets at https://inversescaling.com/data to allow for further investigation of inverse scaling. Our tasks have helped drive the discovery of U-shaped and inverted-U scaling trends, where an initial trend reverses, suggesting that scaling trends are less reliable at predicting the behavior of larger-scale models than previously understood. Overall, our results suggest that there are tasks for which increased model scale alone may not lead to progress, and that more careful thought needs to go into the data and objectives for training language models.