mmBERT：一種具備退火語言學習的現代多語言編碼器

摘要

僅含編碼器的語言模型常被應用於多種標準機器學習任務中，包括分類與檢索。然而，近期針對編碼器模型的研究，尤其是多語言模型方面，顯得相對匱乏。我們在此介紹mmBERT，這是一款僅含編碼器的語言模型，其預訓練基於超過1800種語言、總計3T詞彙量的多語種文本。構建mmBERT過程中，我們引入了多項創新元素，如逆掩碼比率調度與逆溫度採樣比率。我們在衰減階段才將超過1700種低資源語言納入數據混合，結果顯示此舉顯著提升了模型性能，並最大化地利用了相對有限的訓練數據所帶來的增益。儘管這些低資源語言僅在短暫的衰減階段被包含，我們仍達到了與OpenAI的o3及Google的Gemini 2.5 Pro等模型相當的分類性能。總體而言，我們證明了mmBERT在分類與檢索任務上，無論是對高資源還是低資源語言，均顯著超越了前一代模型的表現。

English

Encoder-only languages models are frequently used for a variety of standard machine learning tasks, including classification and retrieval. However, there has been a lack of recent research for encoder models, especially with respect to multilingual models. We introduce mmBERT, an encoder-only language model pretrained on 3T tokens of multilingual text in over 1800 languages. To build mmBERT we introduce several novel elements, including an inverse mask ratio schedule and an inverse temperature sampling ratio. We add over 1700 low-resource languages to the data mix only during the decay phase, showing that it boosts performance dramatically and maximizes the gains from the relatively small amount of training data. Despite only including these low-resource languages in the short decay phase we achieve similar classification performance to models like OpenAI's o3 and Google's Gemini 2.5 Pro. Overall, we show that mmBERT significantly outperforms the previous generation of models on classification and retrieval tasks -- on both high and low-resource languages.

mmBERT：一種具備退火語言學習的現代多語言編碼器

mmBERT: A Modern Multilingual Encoder with Annealed Language Learning

摘要

Support