PointArena：透過語言引導指向探討多模態基礎理解

PointArena: Probing Multimodal Grounding Through Language-Guided Pointing

May 15, 2025

作者: Long Cheng, Jiafei Duan, Yi Ru Wang, Haoquan Fang, Boyang Li, Yushan Huang, Elvis Wang, Ainaz Eftekhar, Jason Lee, Wentao Yuan, Rose Hendrix, Noah A. Smith, Fei Xia, Dieter Fox, Ranjay Krishna

cs.AI

摘要

指向作为一种基础且直观的机制，在视觉语境中为语言提供基础，其应用范围涵盖机器人技术、辅助技术和交互式人工智能系统。尽管最近的多模态模型已开始支持指向功能，但现有的基准测试通常仅关注于指代性物体定位任务。我们引入了PointArena，一个用于评估多模态指向在多样化推理场景中的综合平台。PointArena包含三个组成部分：(1) Point-Bench，一个精心策划的数据集，包含约1,000个指向任务，覆盖五个推理类别；(2) Point-Battle，一个基于网络的互动竞技场，支持盲目的成对模型比较，已收集超过4,500次匿名投票；(3) Point-Act，一个现实世界的机器人操作系统，允许用户在实际环境中直接评估多模态模型的指向能力。我们对最先进的开源和专有多模态模型进行了广泛评估。结果表明，Molmo-72B持续优于其他模型，尽管专有模型逐渐展现出可比的性能。此外，我们发现针对指向任务的有监督训练显著提升了模型性能。在我们的多阶段评估流程中，我们还观察到强烈的相关性，强调了精确指向能力在使多模态模型有效连接抽象推理与具体现实世界行动中的关键作用。项目页面：https://pointarena.github.io/

English

Pointing serves as a fundamental and intuitive mechanism for grounding language within visual contexts, with applications spanning robotics, assistive technologies, and interactive AI systems. While recent multimodal models have started to support pointing capabilities, existing benchmarks typically focus only on referential object localization tasks. We introduce PointArena, a comprehensive platform for evaluating multimodal pointing across diverse reasoning scenarios. PointArena comprises three components: (1) Point-Bench, a curated dataset containing approximately 1,000 pointing tasks across five reasoning categories; (2) Point-Battle, an interactive, web-based arena facilitating blind, pairwise model comparisons, which has already gathered over 4,500 anonymized votes; and (3) Point-Act, a real-world robotic manipulation system allowing users to directly evaluate multimodal model pointing capabilities in practical settings. We conducted extensive evaluations of both state-of-the-art open-source and proprietary multimodal models. Results indicate that Molmo-72B consistently outperforms other models, though proprietary models increasingly demonstrate comparable performance. Additionally, we find that supervised training specifically targeting pointing tasks significantly enhances model performance. Across our multi-stage evaluation pipeline, we also observe strong correlations, underscoring the critical role of precise pointing capabilities in enabling multimodal models to effectively bridge abstract reasoning with concrete, real-world actions. Project page: https://pointarena.github.io/