数据厨师：基于强化学习为大型语言模型定制最优数据配方

摘要

在当前大语言模型（LLM）发展格局中，大规模高质量训练数据的构建是提升模型性能的核心驱动力。数据配方作为关键杠杆，包含将原始数据源转化为训练语料库的完整处理流程。尽管LLM已逐渐应用于自动化执行数据处理环节（如数据合成与过滤），但数据配方的整体设计仍高度依赖人工，需要大量专业知识和反复迭代。为突破这一瓶颈，我们提出了面向LLM领域适应的端到端数据配方生成框架：给定目标基准测试和可用数据源池，模型需输出能够使基础LLM适配目标任务的完整数据配方。我们推出的DataChef-32B模型采用在线强化学习策略，通过代理奖励函数预测候选配方的下游性能。在六项保留任务上的实验表明，DataChef-32B生成的实用配方可实现与专家手工设计配方相当的下游性能。尤为突出的是，该模型生成的数学领域适配配方使Qwen3-1.7B-Base在AIME'25测试中达到66.7分，超越原版Qwen3-1.7B。这项研究为自动化LLM训练及开发自进化AI系统提供了新思路。

English

In the current landscape of Large Language Models (LLMs), the curation of large-scale, high-quality training data is a primary driver of model performance. A key lever is the data recipe, which comprises a data processing pipeline to transform raw sources into training corpora. Despite the growing use of LLMs to automate individual data processing steps, such as data synthesis and filtering, the overall design of data recipes remains largely manual and labor-intensive, requiring substantial human expertise and iteration. To bridge this gap, we formulate end-to-end data recipe generation for LLM adaptation. Given a target benchmark and a pool of available data sources, a model is required to output a complete data recipe that adapts a base LLM to the target task. We present DataChef-32B, which performs online reinforcement learning using a proxy reward that predicts downstream performance for candidate recipes. Across six held-out tasks, DataChef-32B produces practical recipes that reach comparable downstream performance to those curated by human experts. Notably, the recipe from DataChef-32B adapts Qwen3-1.7B-Base to the math domain, achieving 66.7 on AIME'25 and surpassing Qwen3-1.7B. This work sheds new light on automating LLM training and developing self-evolving AI systems.

数据厨师：基于强化学习为大型语言模型定制最优数据配方

DataChef: Cooking Up Optimal Data Recipes for LLM Adaptation via Reinforcement Learning

摘要

Support