MOSAIC：一种辅助和互动烹饪的模块化系统

摘要

我们介绍了MOSAIC，这是一个用于家庭机器人执行复杂协作任务的模块化架构，例如与日常用户一起烹饪。MOSAIC与人类紧密合作，使用自然语言与用户交互，协调多个机器人，并管理日常物品的开放词汇表。在其核心，MOSAIC采用模块化：它利用多个大规模预训练模型来执行通用任务，如语言和图像识别，同时使用为特定任务设计的简化模块进行控制。我们在60个端到端试验中对MOSAIC进行了广泛评估，在这些试验中，两个机器人与一个人类用户合作烹饪6种食谱的组合。我们还对各个模块进行了广泛测试，包括180个视觉动作拾取实验，60个人体运动预测实验，以及46次在线用户对任务规划器的评估。我们展示了MOSAIC能够通过与真实人类用户一起运行整个系统来高效地与人类合作，完成了6种不同食谱的68.3%（41/60）协作烹饪试验，子任务完成率为91.6%。最后，我们讨论了当前系统的局限性以及该领域中令人兴奋的开放挑战。该项目的网站位于https://portal-cornell.github.io/MOSAIC/。

English

We present MOSAIC, a modular architecture for home robots to perform complex collaborative tasks, such as cooking with everyday users. MOSAIC tightly collaborates with humans, interacts with users using natural language, coordinates multiple robots, and manages an open vocabulary of everyday objects. At its core, MOSAIC employs modularity: it leverages multiple large-scale pre-trained models for general tasks like language and image recognition, while using streamlined modules designed for task-specific control. We extensively evaluate MOSAIC on 60 end-to-end trials where two robots collaborate with a human user to cook a combination of 6 recipes. We also extensively test individual modules with 180 episodes of visuomotor picking, 60 episodes of human motion forecasting, and 46 online user evaluations of the task planner. We show that MOSAIC is able to efficiently collaborate with humans by running the overall system end-to-end with a real human user, completing 68.3% (41/60) collaborative cooking trials of 6 different recipes with a subtask completion rate of 91.6%. Finally, we discuss the limitations of the current system and exciting open challenges in this domain. The project's website is at https://portal-cornell.github.io/MOSAIC/

MOSAIC：一种辅助和互动烹饪的模块化系统

MOSAIC: A Modular System for Assistive and Interactive Cooking

摘要

Support