DataPrep-Bench：評測大型語言模型作為訓練數據準備者的基準測試

摘要

訓練資料的品質從根本上決定了大型語言模型的能力，但目前尚無統一的基準能全面衡量大型語言模型、代理程式及資料中心工作流程如何端到端地準備訓練資料。我們將大型語言模型驅動的資料準備視為兩種互補能力的結合：資料建構——將原始來源轉換為監督式訓練資料，以及資料品質評估——在下游訓練前預測候選資料集的訓練價值；此處的「品質」指的是下游訓練實用性，而非表面的文字屬性。我們提出DataPrep-Bench，這是首個統一基準，能在六個領域及多個基礎模型上，基於共享的下游驗證協議，同時評估這兩種能力。在資料建構方面，各方法使用相同的原始來源，並透過將基礎模型在其輸出與Dolly-15k共同微調後的表現進行評分；我們同時釋出Data-Construction-Skill，這是一個技能導向的代理程式，在Llama-3.1-8B金融領域中將僅使用Dolly的基線提升了近20個絕對百分點，且在知識萃取密集領域中與最強的代理程式與基於DataFlow的方法表現相當。在資料品質評估方面，評分函數依據其對共享候選池下游表現的皮爾森相關係數進行評分；我們釋出分布對齊分數（DAS），這是一種基於分布的評估工具，透過候選資料集與領域代理之間的MMD進行評估。DAS在六個領域中的四個達到最強的跨模型相關性，且是唯一在數學、科學與醫學領域同時超越r > 0.70的指標，表現優於現有的品質、多樣性與啟發式評估工具。DataPrep-Bench提供了一個統一的、以下游為基礎的框架，用以衡量這兩種能力（作為大型語言模型驅動資料準備的同等重要目標）的進展。

English

The quality of training data fundamentally determines the capabilities of large language models (LLMs), yet no unified benchmark exists to measure how well LLMs, agents, and data-centric workflows actually prepare training data end to end. We view LLM-driven data preparation as comprising two complementary capabilities: data construction, which transforms raw sources into supervised training data, and data quality evaluation, which predicts the training value of candidate datasets before downstream training; throughout, "quality" refers to downstream training utility rather than surface-level textual properties. We introduce DataPrep-Bench, the first unified benchmark that jointly evaluates both capabilities under a shared downstream-grounded protocol over six domains and multiple base models. For data construction, methods consume identical raw sources and are scored by fine-tuning a base model on their outputs jointly with Dolly-15k; alongside this track we release Data-Construction-Skill, a skill-guided agent that lifts the Dolly-only baseline by nearly 20 points absolute on Llama-3.1-8B Finance and is competitive with the strongest agent- and DataFlow-based methods in knowledge-extraction-dense domains. For data quality evaluation, scoring functions are scored by Pearson correlation with downstream performance on a shared candidate pool; we release the Distributional Alignment Score (DAS), a distribution-based evaluator that uses MMD between a candidate dataset and a domain proxy. DAS attains the strongest cross-model correlation in four of six domains and is the only metric clearing r > 0.70 simultaneously in Math, Science, and Medical, outperforming existing quality-, diversity-, and heuristic-based evaluators. DataPrep-Bench provides a unified, downstream-grounded framework for measuring progress on both capabilities as co-equal targets of LLM-driven data preparation.