FlexOlmo：面向靈活數據使用的開放語言模型

FlexOlmo: Open Language Models for Flexible Data Use

July 9, 2025

作者: Weijia Shi, Akshita Bhagia, Kevin Farhat, Niklas Muennighoff, Pete Walsh, Jacob Morrison, Dustin Schwenk, Shayne Longpre, Jake Poznanski, Allyson Ettinger, Daogao Liu, Margaret Li, Dirk Groeneveld, Mike Lewis, Wen-tau Yih, Luca Soldaini, Kyle Lo, Noah A. Smith, Luke Zettlemoyer, Pang Wei Koh, Hannaneh Hajishirzi, Ali Farhadi, Sewon Min

cs.AI

摘要

我們推出了FlexOlmo，這是一種新型語言模型（LMs），它支持（1）無需數據共享的分佈式訓練，其中不同的模型參數在封閉數據集上獨立訓練，以及（2）數據靈活的推理，這些參數及其相關數據可以在無需進一步訓練的情況下靈活地包含或排除在模型推理之外。FlexOlmo採用了一種專家混合（MoE）架構，其中每個專家都在封閉數據集上獨立訓練，隨後通過一種新的領域感知路由進行整合，而無需任何聯合訓練。FlexOlmo在FlexMix上進行訓練，這是一個我們精心策劃的語料庫，包含公開可用的數據集以及七個特定領域的數據集，代表了封閉集的現實近似。我們評估了多達370億參數（200億活躍）的模型在31個多樣化的下游任務上的表現。我們展示了在公開數據上訓練的通用專家可以有效地與其他數據所有者獨立訓練的專家結合，帶來平均41%的相對改進，同時允許用戶根據數據許可或權限要求選擇退出某些數據。我們的方法在平均上比先前的模型合併方法高出10.1%，並且在相同的訓練FLOPs下，超越了未經數據限制訓練的標準MoE。總的來說，這項研究為擁有敏感或受保護數據的受監管行業的數據所有者和研究人員提供了一種解決方案。FlexOlmo使得能夠從封閉數據中受益，同時通過保持數據本地化並支持推理期間數據訪問的細粒度控制來尊重數據所有者的偏好。

English

We introduce FlexOlmo, a new class of language models (LMs) that supports (1) distributed training without data sharing, where different model parameters are independently trained on closed datasets, and (2) data-flexible inference, where these parameters along with their associated data can be flexibly included or excluded from model inferences with no further training. FlexOlmo employs a mixture-of-experts (MoE) architecture where each expert is trained independently on closed datasets and later integrated through a new domain-informed routing without any joint training. FlexOlmo is trained on FlexMix, a corpus we curate comprising publicly available datasets alongside seven domain-specific sets, representing realistic approximations of closed sets. We evaluate models with up to 37 billion parameters (20 billion active) on 31 diverse downstream tasks. We show that a general expert trained on public data can be effectively combined with independently trained experts from other data owners, leading to an average 41% relative improvement while allowing users to opt out of certain data based on data licensing or permission requirements. Our approach also outperforms prior model merging methods by 10.1% on average and surpasses the standard MoE trained without data restrictions using the same training FLOPs. Altogether, this research presents a solution for both data owners and researchers in regulated industries with sensitive or protected data. FlexOlmo enables benefiting from closed data while respecting data owners' preferences by keeping their data local and supporting fine-grained control of data access during inference.