粗略对应引发多模语言模型中的3D时空理解

摘要

多模态语言模型（MLLMs）越来越多地被应用于现实世界的环境中，这要求它们具备解释3D空间和理解时间动态的能力。尽管具有潜力，但我们社区内目前顶尖的模型仍然在充分理解空间和时间维度方面存在不足。我们引入了粗糙对应（Coarse Correspondence），这是一种简单、无需训练、有效且通用的视觉提示方法，用于引发多模态LLMs对3D和时间的理解。我们的方法使用轻量级跟踪模型在视频的帧之间或图像视角集之间找到物体对应关系。它选择最频繁出现的物体实例，并在图像中用带有唯一ID的标记进行可视化。通过这种简单方法，我们在包括ScanQA（+20.5\%）和OpenEQA的子集（+9.7\%）在内的3D理解基准测试中取得了最先进的结果，并且在诸如EgoSchema（+6.0\%）等长视频基准测试中也取得了优异成绩。我们还整理了一个小型诊断数据集，以评估MLLMs是否能够从除相机视角以外的描述视角推理空间。再次，粗糙对应提高了空间透视能力，但我们强调MLLMs在这项任务上存在困难。综上所述，我们展示了我们的简单提示方法可以显著帮助需要3D或时间推理的下游任务。

English

Multimodal language models (MLLMs) are increasingly being implemented in real-world environments, necessitating their ability to interpret 3D spaces and comprehend temporal dynamics. Despite their potential, current top models within our community still fall short in adequately understanding spatial and temporal dimensions. We introduce Coarse Correspondence, a simple, training-free, effective, and general-purpose visual prompting method to elicit 3D and temporal understanding in multimodal LLMs. Our method uses a lightweight tracking model to find object correspondences between frames in a video or between sets of image viewpoints. It selects the most frequent object instances and visualizes them with markers with unique IDs in the image. With this simple approach, we achieve state-of-the-art results on 3D understanding benchmarks including ScanQA (+20.5\%) and a subset of OpenEQA (+9.7\%), and on long-form video benchmarks such as EgoSchema (+6.0\%). We also curate a small diagnostic dataset to evaluate whether MLLMs can reason about space from a described viewpoint other than the camera viewpoint. Again, Coarse Correspondence improves spatial perspective-taking abilities but we highlight that MLLMs struggle with this task. Together, we demonstrate that our simple prompting method can significantly aid downstream tasks that require 3D or temporal reasoning.

粗略对应引发多模语言模型中的3D时空理解

Coarse Correspondence Elicit 3D Spacetime Understanding in Multimodal Language Model

摘要

Summary

Support

Support