全语言通用模型的未来：OmniBench

OmniBench: Towards The Future of Universal Omni-Language Models

September 23, 2024

作者: Yizhi Li, Ge Zhang, Yinghao Ma, Ruibin Yuan, Kang Zhu, Hangyu Guo, Yiming Liang, Jiaheng Liu, Jian Yang, Siwei Wu, Xingwei Qu, Jinjie Shi, Xinyue Zhang, Zhenzhu Yang, Xiangzhou Wang, Zhaoxiang Zhang, Zachary Liu, Emmanouil Benetos, Wenhao Huang, Chenghua Lin

cs.AI

摘要

最近，多模态大型语言模型（MLLMs）的最新进展旨在整合和解释跨多种形式的数据。然而，这些模型同时处理和推理多种形式的能力仍然未被充分探索，部分原因是缺乏全面的形式化基准。我们引入了OmniBench，这是一个新颖的基准，旨在严格评估模型同时识别、解释和推理视觉、声学和文本输入的能力。我们将能够进行三模态处理的模型定义为全语言模型（OLMs）。OmniBench的独特之处在于具有高质量的人类注释，确保准确的响应需要跨三种形式进行整合理解和推理。我们的主要发现表明：i）开源OLMs在三模态环境中的遵循指令和推理能力存在关键限制；ii）即使为图像和音频提供替代文本表示，基准模型的表现也不佳（低于50％的准确率）。这些结果表明，在现有的MLLM训练范式中，构建文本、图像和音频的一致上下文的能力经常被忽视。我们主张未来的研究应专注于开发更强大的三模态整合技术和训练策略，以提高OLM在多种形式上的性能。代码和实时排行榜可在https://m-a-p.ai/OmniBench找到。

English

Recent advancements in multimodal large language models (MLLMs) have aimed to integrate and interpret data across diverse modalities. However, the capacity of these models to concurrently process and reason about multiple modalities remains inadequately explored, partly due to the lack of comprehensive modality-wise benchmarks. We introduce OmniBench, a novel benchmark designed to rigorously evaluate models' ability to recognize, interpret, and reason across visual, acoustic, and textual inputs simultaneously. We define models capable of such tri-modal processing as omni-language models (OLMs). OmniBench is distinguished by high-quality human annotations, ensuring that accurate responses require integrated understanding and reasoning across all three modalities. Our main findings reveal that: i) open-source OLMs exhibit critical limitations in instruction-following and reasoning capabilities within tri-modal contexts; and ii) the baseline models perform poorly (below 50% accuracy) even when provided with alternative textual representations of images and audio. These results suggest that the ability to construct a consistent context from text, image, and audio is often overlooked in existing MLLM training paradigms. We advocate for future research to focus on developing more robust tri-modal integration techniques and training strategies to enhance OLM performance across diverse modalities. The codes and live leaderboard could be found at https://m-a-p.ai/OmniBench.