AgenticDataBench：数据智能体综合基准测试

摘要

数据科学旨在从异构原始数据中提取可操作的见解，释放现代社会海量数据生成的价值。实现这一过程的自动化对于减少数据科学家的劳动密集型工作、推动可扩展的数据驱动应用至关重要。近年来，基于大语言模型的数据代理已成为自动化数据科学工作流程的有前景的解决方案。然而，该领域缺乏全面的基准测试来以细粒度方式对跨不同场景的这些代理进行严格评估。为填补这一空白，我们提出了AgenticDataBench——一个涵盖多个领域真实任务、带有细粒度真实标签的综合性基准测试。这使评估能够捕捉数据科学工作流程的多样性和复杂性，以及代理的详细性能。首先，为覆盖不同领域，我们从15个垂直行业中收集真实数据集和任务，包括来自一家领先金融科技公司的5个实际B2B用例。其次，为消除真实世界任务中的冗余性，并为缺乏真实数据的领域生成高质量任务，我们引入了数据科学技能（即重复出现的数据中心操作模式），并通过包含的技能数量量化基准覆盖范围。代表性技能是从Stack Overflow上大规模任务解决方案中，通过技能对齐的层次聚类提取的。第三，对于真实商业任务，我们选择在技能组合上多样性最大化的任务-解决方案对，确保对实际场景的广泛覆盖。第四，为尚无真实任务的设备领域生成现实任务，我们提出一种系统化的基于大语言模型的任务生成方法，基于这些技能创建工作流程和任务。最后，我们使用标注的基准测试和开源实验平台评估了最先进的数据代理，提供了详细的技能级洞察。

English

Data science aims to derive actionable insights from heterogeneous raw data, unlocking the value of the massive amounts of data generated in modern society. Automating this process is essential to reducing labor-intensive efforts for data scientists and enabling scalable data-driven applications. Recently, large language model (LLM)-based data agents have emerged as a promising solution to automate data science workflows. However, the field lacks comprehensive benchmarks to rigorously evaluate these agents across diverse scenarios with fine-grained granularity. To address this gap, we propose AgenticDataBench, a comprehensive benchmark featuring realistic tasks spanning diverse domains with fine-grained ground-truth labels. This enables evaluations to capture the diversity and complexity of data science workflows and the detailed performance of agents. First, to cover diverse domains, we collect real datasets and tasks from 15 vertical domains, including 5 real-world B2B use cases from a leading fintech company. Second, to remove redundancy in real-world tasks and generate high-quality tasks for domains lacking real data, we introduce data science skills, recurring data-centric operational patterns, and quantify benchmark coverage by the number of skills included. Representative skills are extracted from large-scale task solutions on Stack Overflow using skill-aligned hierarchical clustering. Third, for real-world business tasks, we select task-solution pairs that maximize diversity in skill composition, ensuring broad coverage of practical scenarios. Fourth, to generate realistic tasks for devise domains without real tasks, we propose a systematic LLM-based task generation approach to create workflows and tasks based on these skills. Finally, we evaluate state-of-the-art data agents using our annotated benchmark and open-sourced testbed, providing detailed skill-level insights.