PointArena：通过语言引导指向探索多模态基础

PointArena: Probing Multimodal Grounding Through Language-Guided Pointing

May 15, 2025

作者: Long Cheng, Jiafei Duan, Yi Ru Wang, Haoquan Fang, Boyang Li, Yushan Huang, Elvis Wang, Ainaz Eftekhar, Jason Lee, Wentao Yuan, Rose Hendrix, Noah A. Smith, Fei Xia, Dieter Fox, Ranjay Krishna

cs.AI

摘要

指向作为一种基础且直观的机制，在视觉语境中为语言提供锚定，其应用遍及机器人技术、辅助技术和交互式人工智能系统。尽管最近的多模态模型已开始支持指向功能，但现有基准测试通常仅聚焦于指代性物体定位任务。我们推出了PointArena，一个全面评估多模态指向在多样化推理场景中表现的平台。PointArena包含三个组成部分：(1) Point-Bench，一个精心策划的数据集，涵盖五个推理类别下约1,000项指向任务；(2) Point-Battle，一个基于网络的互动竞技场，支持盲测、成对模型比较，已收集超过4,500次匿名投票；(3) Point-Act，一个现实世界中的机器人操作系统，允许用户直接评估多模态模型在实际环境中的指向能力。我们对当前最先进的开源及专有多模态模型进行了广泛评估。结果显示，Molmo-72B持续优于其他模型，尽管专有模型逐渐展现出与之相当的性能。此外，我们发现专门针对指向任务的有监督训练显著提升了模型表现。在我们的多阶段评估流程中，我们还观察到强烈的相关性，这突显了精确指向能力在使多模态模型有效连接抽象推理与具体现实世界行动中的关键作用。项目页面：https://pointarena.github.io/

English

Pointing serves as a fundamental and intuitive mechanism for grounding language within visual contexts, with applications spanning robotics, assistive technologies, and interactive AI systems. While recent multimodal models have started to support pointing capabilities, existing benchmarks typically focus only on referential object localization tasks. We introduce PointArena, a comprehensive platform for evaluating multimodal pointing across diverse reasoning scenarios. PointArena comprises three components: (1) Point-Bench, a curated dataset containing approximately 1,000 pointing tasks across five reasoning categories; (2) Point-Battle, an interactive, web-based arena facilitating blind, pairwise model comparisons, which has already gathered over 4,500 anonymized votes; and (3) Point-Act, a real-world robotic manipulation system allowing users to directly evaluate multimodal model pointing capabilities in practical settings. We conducted extensive evaluations of both state-of-the-art open-source and proprietary multimodal models. Results indicate that Molmo-72B consistently outperforms other models, though proprietary models increasingly demonstrate comparable performance. Additionally, we find that supervised training specifically targeting pointing tasks significantly enhances model performance. Across our multi-stage evaluation pipeline, we also observe strong correlations, underscoring the critical role of precise pointing capabilities in enabling multimodal models to effectively bridge abstract reasoning with concrete, real-world actions. Project page: https://pointarena.github.io/