InstructPart：基于指令推理的任务导向型部件分割

摘要

大型多模态基础模型，尤其在语言与视觉领域，已显著推动了包括机器人技术、自动驾驶、信息检索及场景理解在内的多项任务进展。然而，众多此类模型将物体视为不可分割的整体，忽视了构成物体的各个部件。理解这些部件及其相关的功能属性，对于深入掌握物体的功能性至关重要，这是执行广泛任务的基础。在本研究中，我们引入了一个新颖的现实世界基准——InstructPart，它包含了手工标注的部件分割注释及面向任务的指令，旨在评估现有模型在日常情境下理解和执行部件级任务的能力。通过实验，我们证明了即使对于最先进的视觉-语言模型（VLMs），面向任务的部件分割仍是一个具有挑战性的问题。除了我们的基准外，我们还提出了一个简单基线模型，通过使用我们的数据集进行微调，实现了性能的翻倍提升。借助我们的数据集与基准，我们期望能促进面向任务的部件分割研究，并提升VLMs在机器人技术、虚拟现实、信息检索及其他相关领域的应用潜力。项目网站：https://zifuwan.github.io/InstructPart/。

English

Large multimodal foundation models, particularly in the domains of language and vision, have significantly advanced various tasks, including robotics, autonomous driving, information retrieval, and grounding. However, many of these models perceive objects as indivisible, overlooking the components that constitute them. Understanding these components and their associated affordances provides valuable insights into an object's functionality, which is fundamental for performing a wide range of tasks. In this work, we introduce a novel real-world benchmark, InstructPart, comprising hand-labeled part segmentation annotations and task-oriented instructions to evaluate the performance of current models in understanding and executing part-level tasks within everyday contexts. Through our experiments, we demonstrate that task-oriented part segmentation remains a challenging problem, even for state-of-the-art Vision-Language Models (VLMs). In addition to our benchmark, we introduce a simple baseline that achieves a twofold performance improvement through fine-tuning with our dataset. With our dataset and benchmark, we aim to facilitate research on task-oriented part segmentation and enhance the applicability of VLMs across various domains, including robotics, virtual reality, information retrieval, and other related fields. Project website: https://zifuwan.github.io/InstructPart/.

InstructPart：基于指令推理的任务导向型部件分割

InstructPart: Task-Oriented Part Segmentation with Instruction Reasoning

摘要

Summary

Support

Support