从零开始扩展原生多模态预训练

摘要

尽管大型语言模型（LLMs）展现出卓越的推理能力，但其对纯文本预训练的依赖限制了其对多模态物理世界的感知。原生多模态预训练通过从零开始在多模态输入上进行训练，避免了这一局限，从而实现深层次的跨模态整合，并缓解了传统后期融合架构中固有的优化不对称性。尽管具有这些优势，该范式的缩放属性仍缺乏系统性表征。为弥补这一空白，我们探讨了在固定计算预算下训练基于Transformer的视觉-语言模型时，最优模型规模与token数量的关系。研究表明，最小化目标损失遵循可预测的计算定律，而计算最优的模型规模与token数量呈现幂律缩放规律。值得注意的是，语言目标与多模态目标展现出不同的缩放行为：语言分配律对数据组成基本不敏感，表明无论多模态数据比例如何变化，语言学习均保持稳定；而多模态分配律则对数据组成高度敏感。具体而言，文本密集数据仅在更大模型规模下才能实现计算效率，这迫使最优资源配置向更大模型容量倾斜。此外，通过建模数据组成对计算定律和分配指数的影响，我们推导出效率前沿曲线，明确了模型规模、token数量与数据混合比例的具体配置。下游评估进一步揭示，原生多模态预训练会引发正向跨模态迁移，从而增强纯文本空间推理能力，并实现稳健的多模态上下文学习。综上所述，本实证研究为可预测地缩放多模态基础模型奠定了关键基础。

English

Although large language models (LLMs) exhibit remarkable reasoning capabilities, their reliance on text-only pre-training restricts the perception of the multimodal physical world. Native multimodal pre-training avoids this limitation by training models from scratch on multimodal inputs, thereby achieving deep cross-modal integration and mitigating optimization asymmetries inherent to traditional late-fusion architectures. Despite these advantages, the scaling properties of this paradigm remain systematically uncharacterized. To address this gap, we investigate the optimal model size and token count for training a transformer-based vision-language model under a fixed computational budget. We demonstrate that minimal objective loss adheres to a predictable compute law, whereas compute-optimal model sizes and token counts scale as power laws. Notably, language and multimodal objectives manifest distinct scaling behaviors. The language allocation law is largely invariant to the composition of the data, indicating stable language learning regardless of the multimodal data ratio. Conversely, the multimodal allocation law is highly sensitive to this composition. Specifically, text-heavy mixtures become compute-efficient only at larger model scales, shifting the optimal resource allocation toward greater model capacity. Additionally, by modeling the influence of data composition on compute laws and allocation exponents, we derive an efficiency frontier specifying precise configurations of model size, token count, and data mixture. Downstream evaluations further reveal that native multimodal pre-training induces positive cross-modal transfer, thereby enhancing pure-text spatial reasoning and enabling robust multimodal in-context learning. In summary, this empirical research establishes the essential groundwork for predictably scaling multimodal foundation models.