從零開始擴展原生多模態預訓練

摘要

儘管大型語言模型展現了卓越的推理能力，但其依賴純文本預訓練的限制，使其對多模態物理世界的感知有所不足。原生多模態預訓練透過從頭開始以多模態輸入訓練模型，避免了此限制，從而實現深層跨模態整合，並緩解傳統後期融合架構中固有的優化不對稱性。儘管具備這些優勢，該範式的縮放特性仍未獲得系統性表徵。為填補此研究缺口，我們探討在固定計算預算下，訓練基於Transformer的視覺語言模型時的最優模型規模與詞元數量。我們證明了最小目標損失遵循可預測的計算法則，而計算最優的模型規模與詞元數量則按冪次法則縮放。值得注意的是，語言目標與多模態目標展現出不同的縮放行為。語言分配法則對數據組成大致保持不變，顯示無論多模態數據比例如何，語言學習過程均維持穩定。相反，多模態分配法則對數據組成極其敏感。具體而言，文本密集型混合僅在較大模型規模下才能實現計算效率，從而將最優資源分配轉向更大的模型容量。此外，透過對數據組成影響計算法則與分配指數進行建模，我們推導出效率邊界，明確指定了模型規模、詞元數量與數據混合比例的精確配置。下游評測進一步揭示，原生多模態預訓練可誘發正向跨模態遷移，從而增強純文本空間推理能力，並實現穩健的多模態情境學習。總而言之，這項實證研究為可預測地縮放多模態基礎模型奠定了重要基礎。

English

Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world. Native multimodal pre-training avoids this limitation by training models from scratch on multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain systematically uncharacterized. To address this gap, we investigate the optimal model size and token count for training a transformer-based vision-language model under a fixed computational budget. We demonstrate that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives manifest distinct scaling behaviors. The language allocation law is largely invariant to the composition of the data, indicating stable language learning regardless of the multimodal data ratio. Conversely, the multimodal allocation law is highly sensitive to this composition. Specifically, text-heavy mixtures become compute-efficient only at larger model scales, shifting the optimal resource allocation toward greater model capacity. Additionally, by modeling the influence of data composition on compute laws and allocation exponents, we derive an efficiency frontier specifying precise configurations of model size, token count, and data mixture. Downstream evaluations further reveal that native multimodal pre-training induces positive cross-modal transfer, thereby enhancing pure-text spatial reasoning and enabling robust multimodal in-context learning. In summary, this empirical research establishes the essential groundwork for predictably scaling multimodal foundation models.