SPHINX-X：針對一系列多模式大型語言模型的資料和參數進行擴展。

摘要

我們提出了SPHINX-X，這是一個基於SPHINX開發的廣泛多模式大型語言模型（MLLM）系列。為了改善架構和訓練效率，我們修改了SPHINX框架，去除了多餘的視覺編碼器，通過跳過完全填充的子圖像並使用跳過標記，並將多階段訓練簡化為單階段的全方位範式。為了充分發揮MLLM的潛力，我們匯集了一個包含語言、視覺和視覺語言任務的全面多領域和多模式數據集，覆蓋了公開可用的資源。我們進一步通過我們的OCR密集和Set-of-Mark數據集豐富了這個收藏，擴展了多樣性和普遍性。通過對不同基礎LLM（包括TinyLlama1.1B、InternLM2-7B、LLaMA2-13B和Mixtral8x7B）進行訓練，我們獲得了一系列在參數大小和多語言能力上有所不同的MLLM。全面的基準測試顯示了多模式性能與數據和參數規模之間的強相關性。代碼和模型已在https://github.com/Alpha-VLLM/LLaMA2-Accessory 釋出。

English

We propose SPHINX-X, an extensive Multimodality Large Language Model (MLLM) series developed upon SPHINX. To improve the architecture and training efficiency, we modify the SPHINX framework by removing redundant visual encoders, bypassing fully-padded sub-images with skip tokens, and simplifying multi-stage training into a one-stage all-in-one paradigm. To fully unleash the potential of MLLMs, we assemble a comprehensive multi-domain and multimodal dataset covering publicly available resources in language, vision, and vision-language tasks. We further enrich this collection with our curated OCR intensive and Set-of-Mark datasets, extending the diversity and generality. By training over different base LLMs including TinyLlama1.1B, InternLM2-7B, LLaMA2-13B, and Mixtral8x7B, we obtain a spectrum of MLLMs that vary in parameter size and multilingual capabilities. Comprehensive benchmarking reveals a strong correlation between the multi-modal performance with the data and parameter scales. Code and models are released at https://github.com/Alpha-VLLM/LLaMA2-Accessory

SPHINX-X：針對一系列多模式大型語言模型的資料和參數進行擴展。

SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models

摘要

Support