DataPrep-Bench：将大语言模型作为训练数据准备器的基准测试

摘要

训练数据的质量从根本上决定了大型语言模型（LLM）的能力，但目前尚无统一的基准来衡量LLM、智能体以及数据驱动工作流在端到端准备训练数据方面的表现。我们认为，LLM驱动的数据准备包含两种互补的能力：数据构建——将原始数据源转化为监督训练数据；以及数据质量评估——在下游训练之前预测候选数据集的训练价值。在此过程中，“质量”指下游训练效用，而非表面文本属性。我们提出了DataPrep-Bench，这是首个在六个领域和多个基础模型上，采用统一的下游锚定协议联合评估这两种能力的基准。在数据构建方面，各种方法使用相同的原始数据源，并通过对模型输出与Dolly-15k联合微调基础模型来评分；与此同时，我们发布了Data-Construction-Skill，一个技能引导的智能体，它使Llama-3.1-8B在金融领域的纯Dolly基线提升了近20个绝对百分点，并在知识提取密集的领域与最先进的基于智能体和DataFlow的方法竞争。在数据质量评估方面，评分函数通过与共享候选池中下游性能的皮尔逊相关性来评分；我们发布了分布对齐分数（DAS），这是一种基于分布的评估器，使用候选数据集与领域代理之间的最大均值差异（MMD）。DAS在六个领域中的四个取得了最强的跨模型相关性，且是唯一在数学、科学和医学领域同时达到r > 0.70的指标，优于现有的基于质量、多样性和启发式的评估器。DataPrep-Bench提供了一个统一的、下游锚定的框架，用于衡量这两种能力作为LLM驱动数据准备的同等目标的进展。

English

The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data end to end. We view LLM-driven data preparation as comprising two complementary capabilities: data construction, which transforms raw sources into supervised training data, and data quality evaluation, which predicts the training value of candidate datasets before downstream training; throughout, "quality" refers to downstream training utility rather than surface-level textual properties. We introduce DataPrep-Bench, the first unified benchmark that jointly evaluates both capabilities under a shared downstream-grounded protocol over six domains and multiple base models. For data construction, methods consume identical raw sources and are scored by fine-tuning a base model on their outputs jointly with Dolly-15k; alongside this track we release Data-Construction-Skill, a skill-guided agent that lifts the Dolly-only baseline by nearly 20 points absolute on Llama-3.1-8B Finance and is competitive with the strongest agent- and DataFlow-based methods in knowledge-extraction-dense domains. For data quality evaluation, scoring functions are scored by Pearson correlation with downstream performance on a shared candidate pool; we release the Distributional Alignment Score (DAS), a distribution-based evaluator that uses MMD between a candidate dataset and a domain proxy. DAS attains the strongest cross-model correlation in four of six domains and is the only metric clearing r > 0.70 simultaneously in Math, Science, and Medical, outperforming existing quality-, diversity-, and heuristic-based evaluators. DataPrep-Bench provides a unified, downstream-grounded framework for measuring progress on both capabilities as co-equal targets of LLM-driven data preparation.