MADLAD-400：一個多語言且文件級別的大型審核數據集
MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
September 9, 2023
作者: Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Christopher A. Choquette-Choo, Katherine Lee, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, Orhan Firat
cs.AI
摘要
我們介紹了MADLAD-400，這是一個手動審核的通用領域3T令牌單語數據集，基於CommonCrawl，涵蓋了419種語言。我們討論了自我審核MADLAD-400揭示的限制，以及數據審核在數據集創建過程中的作用。然後，我們使用公開可用數據訓練並發布了一個包含107億參數的多語言機器翻譯模型，覆蓋了超過450種語言的2500億令牌，發現它與大得多的模型相競爭，並在不同領域報告結果。此外，我們訓練了一個包含80億參數的語言模型，並對少樣本翻譯結果進行評估。我們將基準模型提供給研究社區。
English
We introduce MADLAD-400, a manually audited, general domain 3T token
monolingual dataset based on CommonCrawl, spanning 419 languages. We discuss
the limitations revealed by self-auditing MADLAD-400, and the role data
auditing had in the dataset creation process. We then train and release a
10.7B-parameter multilingual machine translation model on 250 billion tokens
covering over 450 languages using publicly available data, and find that it is
competitive with models that are significantly larger, and report the results
on different domains. In addition, we train a 8B-parameter language model, and
assess the results on few-shot translation. We make the baseline models
available to the research community.