M3DBench：使用多模式3D提示指導大型模型

摘要

最近，3D理解已變得流行，以促進自主代理進行進一步的決策。然而，現有的3D數據集和方法通常局限於特定任務。另一方面，大型語言模型（LLMs）和多模式語言模型（MLMs）的最新進展展示了出色的通用語言和圖像任務表現。因此，將MLM的潛力解鎖為更廣泛任務的3D通用人才是一個有趣的方向。然而，由於缺乏大規模的3D指示跟隨數據集，目前MLMs的研究對3D任務的關注較少。在這項工作中，我們介紹了一個名為M3DBench的全面3D指示跟隨數據集，具有以下特點：1）它支持與文本、圖像、3D對象和其他視覺提示交錯的通用多模式指令。2）它統一了不同區域和場景級別的多樣化3D任務，涵蓋了現實世界3D環境中的各種基本能力。3）它是一個擁有超過320k指示-回應對的大規模3D指示跟隨數據集。此外，我們建立了一個新的基準來評估大型模型在理解多模式3D提示方面的性能。廣泛的實驗證明了我們數據集和基準線的有效性，支持通用的3D中心任務，這可能激發未來的研究。

English

Recently, 3D understanding has become popular to facilitate autonomous agents to perform further decisionmaking. However, existing 3D datasets and methods are often limited to specific tasks. On the other hand, recent progress in Large Language Models (LLMs) and Multimodal Language Models (MLMs) have demonstrated exceptional general language and imagery tasking performance. Therefore, it is interesting to unlock MLM's potential to be 3D generalist for wider tasks. However, current MLMs' research has been less focused on 3D tasks due to a lack of large-scale 3D instruction-following datasets. In this work, we introduce a comprehensive 3D instructionfollowing dataset called M3DBench, which possesses the following characteristics: 1) It supports general multimodal instructions interleaved with text, images, 3D objects, and other visual prompts. 2) It unifies diverse 3D tasks at both region and scene levels, covering a variety of fundamental abilities in real-world 3D environments. 3) It is a large-scale 3D instruction-following dataset with over 320k instruction-response pairs. Furthermore, we establish a new benchmark for assessing the performance of large models in understanding multi-modal 3D prompts. Extensive experiments demonstrate the effectiveness of our dataset and baseline, supporting general 3D-centric tasks, which can inspire future research.

M3DBench：使用多模式3D提示指導大型模型

M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts

摘要

Support