SeededGrasp：面向多具身複雜場景的語言引導抓取

摘要

在复杂场景中进行实用的机器人抓取，既需要三维空间推理能力，也需要与具体任务要求保持一致。视觉语言模型提供了一种自然的方式，通过语言来指定这些要求，但现有方法要么直接使用视觉语言模型预测抓取位置并受限于有限的空间感知能力，要么将视觉语言模型与抓取模型联合训练，这需要显著更多的数据和计算资源。这些局限性阻碍了性能提升，并使得该方法难以扩展到复杂场景中的多形态机器人。为解决这一问题，我们提出了SeededGrasp，一种新颖的数据高效框架，使视觉语言模型能够预测一个种子点，并将其作为条件输入给后续轻量级的抓取生成模型。我们的架构将高层语义推理与低层几何执行解耦，在避免昂贵端到端训练的同时，支持多形态机器人。为了训练此类模型，我们发布了首个多形态桌面抓取数据集，其中包含杂乱场景中超过250万个抓取点。实验结果表明，我们的方法优于现有基线，在仿真实验中达到72%的成功率，在实际抓取实验中达到78%的成功率。数据和代码详见我们的项目网站：https://uoft-isl.github.io/seeded-grasp/

English

Practical robotic grasping in complex scenes requires both 3D spatial reasoning and alignment with task-specific requirements. Vision-language models (VLMs) offer a natural way to specify these requirements using language, but existing approaches either use a VLM to predict the grasp directly with limited spatial awareness, or train the VLM together with the grasping model, which requires significantly more data and compute. These limitations impede performance and have prevented scaling to multiple embodiments in complex scenes. We address this by proposing SeededGrasp, a novel data-efficient framework that enables a VLM to predict a seed point to be used as conditioning for a subsequent lightweight grasp-generation model. Our architecture decouples high-level semantic reasoning from low-level geometric execution, enabling multi-embodiment support while bypassing the need for expensive end-to-end training. To enable training such models, we release the first multi-embodiment tabletop grasping dataset comprising over 2.5M grasps in cluttered scenes. Experimental results demonstrate that our approach outperforms existing baselines, achieving 72% success in simulation and 78% in real-world grasping experiments. See our project site for data and code: https://uoft-isl.github.io/seeded-grasp/