从文字到数字：您的大型语言模型在提供上下文示例时暗中是一个能干的回归器

From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples

April 11, 2024

作者: Robert Vacareanu, Vlad-Andrei Negru, Vasile Suciu, Mihai Surdeanu

cs.AI

摘要

我们分析了预训练的大型语言模型（例如Llama2、GPT-4、Claude 3等）在提供上下文示例时，无需任何额外训练或梯度更新即可进行线性和非线性回归的表现。我们的研究发现，一些大型语言模型（如GPT-4、Claude 3）能够执行回归任务，其性能与传统监督方法（如随机森林、装袋法或梯度提升）不相上下，甚至表现更好。例如，在具有挑战性的Friedman #2回归数据集上，Claude 3的表现优于许多监督方法，如AdaBoost、支持向量机（SVM）、随机森林、KNN或梯度提升。然后，我们调查了大型语言模型的性能如何随着上下文示例数量的增加而扩展。我们借鉴了在线学习中的遗憾概念，并凭经验证明，大型语言模型能够获得次线性的遗憾。

English

We analyze how well pre-trained large language models (e.g., Llama2, GPT-4, Claude 3, etc) can do linear and non-linear regression when given in-context examples, without any additional training or gradient updates. Our findings reveal that several large language models (e.g., GPT-4, Claude 3) are able to perform regression tasks with a performance rivaling (or even outperforming) that of traditional supervised methods such as Random Forest, Bagging, or Gradient Boosting. For example, on the challenging Friedman #2 regression dataset, Claude 3 outperforms many supervised methods such as AdaBoost, SVM, Random Forest, KNN, or Gradient Boosting. We then investigate how well the performance of large language models scales with the number of in-context exemplars. We borrow from the notion of regret from online learning and empirically show that LLMs are capable of obtaining a sub-linear regret.