GPT-4是一个优秀的数据分析员吗？

Is GPT-4 a Good Data Analyst?

May 24, 2023

作者: Liying Cheng, Xingxuan Li, Lidong Bing

cs.AI

摘要

随着大型语言模型（LLMs）展示出在许多领域和任务中的强大能力，包括上下文理解、代码生成、语言生成、数据叙事等，许多数据分析师可能会担心他们的工作会被人工智能取代。这个有争议的话题引起了公众的广泛关注。然而，我们仍处于意见分歧的阶段，没有得出明确结论。受此启发，我们在本研究中提出了一个研究问题：“GPT-4是否是一名优秀的数据分析师？”并旨在通过进行一对一的比较研究来回答这个问题。具体而言，我们将GPT-4视为一名数据分析师，利用来自各个领域的数据库进行端到端数据分析。我们提出了一个框架来解决这些问题，通过精心设计GPT-4的提示来进行实验。我们还设计了几个任务特定的评估指标，以系统地比较几位专业的人类数据分析师和GPT-4之间的表现。实验结果表明，GPT-4可以达到与人类可比较的表现。我们还对结果进行了深入讨论，以启示进一步研究，然后得出结论：GPT-4可以取代数据分析师。

English

As large language models (LLMs) have demonstrated their powerful capabilities in plenty of domains and tasks, including context understanding, code generation, language generation, data storytelling, etc., many data analysts may raise concerns if their jobs will be replaced by AI. This controversial topic has drawn a lot of attention in public. However, we are still at a stage of divergent opinions without any definitive conclusion. Motivated by this, we raise the research question of "is GPT-4 a good data analyst?" in this work and aim to answer it by conducting head-to-head comparative studies. In detail, we regard GPT-4 as a data analyst to perform end-to-end data analysis with databases from a wide range of domains. We propose a framework to tackle the problems by carefully designing the prompts for GPT-4 to conduct experiments. We also design several task-specific evaluation metrics to systematically compare the performance between several professional human data analysts and GPT-4. Experimental results show that GPT-4 can achieve comparable performance to humans. We also provide in-depth discussions about our results to shed light on further studies before we reach the conclusion that GPT-4 can replace data analysts.