AgenticDataBench：一個全面的數據代理基準

摘要

數據科學旨在從異質原始數據中萃取出可操作的洞見，從而釋放現代社會所產生大量數據的價值。自動化此流程對於減輕數據科學家的人力密集工作、並實現可擴展的數據驅動應用至關重要。近年來，基於大型語言模型的數據代理已成為自動化數據科學工作流程的可行解決方案。然而，該領域缺乏全面的基準測試，無法以細粒度嚴格評估這些代理在不同場景下的表現。為填補此缺口，我們提出 AgenticDataBench——一個全面的基準測試，包含來自多樣領域的實際任務，並提供細粒度的真實標籤。如此一來，評估能夠捕捉數據科學工作流程的多樣性與複雜性，以及代理的詳細表現。首先，為涵蓋多樣領域，我們從15個垂直領域中收集真實數據集與任務，其中包括來自一家領先金融科技公司的5個實際B2B應用案例。其次，為去除實際任務中的冗餘，並為缺乏真實數據的領域生成高品質任務，我們引入數據科學技能——即重複出現的數據中心操作模式——並透過所包含的技能數量來量化基準測試的覆蓋範圍。我們利用技能對齊的階層式聚類，從 Stack Overflow 上的大規模任務解決方案中提取代表性技能。第三，針對實際商業任務，我們選取技能組成多樣性最高的任務—解決方案配對，確保廣泛涵蓋實際場景。第四，為那些缺乏實際任務的設計領域生成具現實感的任務，我們提出一套系統性的基於大型語言模型的任務生成方法，根據這些技能創建工作流程與任務。最後，我們使用已標註的基準測試與開源測試平台，評估當前最先進的數據代理，並提供詳細的技能層級洞見。

English

Data science aims to derive actionable insights from heterogeneous raw data, unlocking the value of the massive amounts of data generated in modern society. Automating this process is essential to reducing labor-intensive efforts for data scientists and enabling scalable data-driven applications. Recently, large language model (LLM)-based data agents have emerged as a promising solution to automate data science workflows. However, the field lacks comprehensive benchmarks to rigorously evaluate these agents across diverse scenarios with fine-grained granularity. To address this gap, we propose AgenticDataBench, a comprehensive benchmark featuring realistic tasks spanning diverse domains with fine-grained ground-truth labels. This enables evaluations to capture the diversity and complexity of data science workflows and the detailed performance of agents. First, to cover diverse domains, we collect real datasets and tasks from 15 vertical domains, including 5 real-world B2B use cases from a leading fintech company. Second, to remove redundancy in real-world tasks and generate high-quality tasks for domains lacking real data, we introduce data science skills, recurring data-centric operational patterns, and quantify benchmark coverage by the number of skills included. Representative skills are extracted from large-scale task solutions on Stack Overflow using skill-aligned hierarchical clustering. Third, for real-world business tasks, we select task-solution pairs that maximize diversity in skill composition, ensuring broad coverage of practical scenarios. Fourth, to generate realistic tasks for devise domains without real tasks, we propose a systematic LLM-based task generation approach to create workflows and tasks based on these skills. Finally, we evaluate state-of-the-art data agents using our annotated benchmark and open-sourced testbed, providing detailed skill-level insights.