SeededGrasp：复杂场景中多种具身形态下的语言引导抓取

摘要

复杂场景下的实用机器人抓取需要同时具备3D空间推理能力和任务特定需求的匹配能力。视觉语言模型（VLM）提供了一种通过语言描述这些需求的自然方式，但现有方法要么直接使用VLM预测抓取动作并受限于空间感知能力不足，要么将VLM与抓取模型联合训练，这需要显著更多的数据和计算量。这些限制阻碍了性能提升，并导致无法扩展到复杂场景中的多实体应用。针对这一问题，我们提出SeededGrasp——一种数据高效的新型框架，使VLM能够预测种子点，并将其作为条件输入后续轻量级抓取生成模型。我们的架构将高层语义推理与低层几何执行解耦，在避免昂贵端到端训练的同时支持多实体应用。为训练此类模型，我们发布了首个多实体桌面抓取数据集，包含杂乱场景中超过250万组抓取数据。实验结果表明，我们的方法优于现有基线方法，在仿真和真实世界抓取实验中分别达到72%和78%的成功率。详见项目网站获取数据与代码：https://uoft-isl.github.io/seeded-grasp/

English

Practical robotic grasping in complex scenes requires both 3D spatial reasoning and alignment with task-specific requirements. Vision-language models (VLMs) offer a natural way to specify these requirements using language, but existing approaches either use a VLM to predict the grasp directly with limited spatial awareness, or train the VLM together with the grasping model, which requires significantly more data and compute. These limitations impede performance and have prevented scaling to multiple embodiments in complex scenes. We address this by proposing SeededGrasp, a novel data-efficient framework that enables a VLM to predict a seed point to be used as conditioning for a subsequent lightweight grasp-generation model. Our architecture decouples high-level semantic reasoning from low-level geometric execution, enabling multi-embodiment support while bypassing the need for expensive end-to-end training. To enable training such models, we release the first multi-embodiment tabletop grasping dataset comprising over 2.5M grasps in cluttered scenes. Experimental results demonstrate that our approach outperforms existing baselines, achieving 72% success in simulation and 78% in real-world grasping experiments. See our project site for data and code: https://uoft-isl.github.io/seeded-grasp/