語言模型合成數據的最佳實踐和經驗教訓

Best Practices and Lessons Learned on Synthetic Data for Language Models

April 11, 2024

作者: Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, Andrew M. Dai

cs.AI

摘要

人工智慧模型的成功取決於大量、多樣且高質量的數據集的可用性，然而由於數據稀缺、隱私問題和高成本，這些數據集往往難以獲得。合成數據已被提出作為一種有前途的解決方案，通過生成模擬真實世界模式的人造數據。本文概述了合成數據研究，討論了其應用、挑戰和未來方向。我們提供了來自先前研究的實證證據，以證明其有效性，並強調確保其真實性、忠實性和無偏見性的重要性。我們強調了對合成數據的負責任使用的需求，以構建更強大、包容和值得信賴的語言模型。

English

The success of AI models relies on the availability of large, diverse, and high-quality datasets, which can be challenging to obtain due to data scarcity, privacy concerns, and high costs. Synthetic data has emerged as a promising solution by generating artificial data that mimics real-world patterns. This paper provides an overview of synthetic data research, discussing its applications, challenges, and future directions. We present empirical evidence from prior art to demonstrate its effectiveness and highlight the importance of ensuring its factuality, fidelity, and unbiasedness. We emphasize the need for responsible use of synthetic data to build more powerful, inclusive, and trustworthy language models.