语言模型的合成数据最佳实践和经验教训

Best Practices and Lessons Learned on Synthetic Data for Language Models

April 11, 2024

作者: Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, Andrew M. Dai

cs.AI

摘要

AI 模型的成功取决于大量、多样化和高质量的数据集的可用性，由于数据稀缺、隐私问题和高成本，获取这些数据集可能具有挑战性。合成数据作为一种有前途的解决方案出现，通过生成模拟真实世界模式的人工数据。本文概述了合成数据研究，讨论了其应用、挑战和未来方向。我们提供了来自先前研究的经验证据，以展示其有效性，并强调确保其真实性、忠实性和无偏见性的重要性。我们强调了对合成数据的负责任使用，以构建更强大、包容和值得信赖的语言模型的必要性。

English

The success of AI models relies on the availability of large, diverse, and high-quality datasets, which can be challenging to obtain due to data scarcity, privacy concerns, and high costs. Synthetic data has emerged as a promising solution by generating artificial data that mimics real-world patterns. This paper provides an overview of synthetic data research, discussing its applications, challenges, and future directions. We present empirical evidence from prior art to demonstrate its effectiveness and highlight the importance of ensuring its factuality, fidelity, and unbiasedness. We emphasize the need for responsible use of synthetic data to build more powerful, inclusive, and trustworthy language models.