MOSAIC：一個模塊化的輔助和互動烹飪系統

摘要

我們提出了 MOSAIC，一種模塊化架構，用於家用機器人執行複雜的協作任務，例如與日常用戶一起烹飪。MOSAIC與人類緊密協作，使用自然語言與用戶互動，協調多個機器人，並管理日常物品的開放詞彙。在其核心，MOSAIC採用模塊化：它利用多個大規模預訓練模型進行一般任務，如語言和圖像識別，同時使用為特定任務設計的精簡模塊進行控制。我們在60個端到端試驗中對MOSAIC進行了廣泛評估，其中兩個機器人與一名人類用戶合作烹飪6種食譜的組合。我們還對個別模塊進行了廣泛測試，包括180個視覺運動撿取情節，60個人體運動預測情節，以及46個任務計劃器的在線用戶評估。我們展示了MOSAIC能夠通過與真實人類用戶一起運行整個系統端到端，有效地與人類協作，完成了6種不同食譜的68.3%（41/60）協作烹飪試驗，子任務完成率為91.6%。最後，我們討論了當前系統的限制以及這一領域中令人興奮的開放挑戰。該項目的網站位於https://portal-cornell.github.io/MOSAIC/。

English

We present MOSAIC, a modular architecture for home robots to perform complex collaborative tasks, such as cooking with everyday users. MOSAIC tightly collaborates with humans, interacts with users using natural language, coordinates multiple robots, and manages an open vocabulary of everyday objects. At its core, MOSAIC employs modularity: it leverages multiple large-scale pre-trained models for general tasks like language and image recognition, while using streamlined modules designed for task-specific control. We extensively evaluate MOSAIC on 60 end-to-end trials where two robots collaborate with a human user to cook a combination of 6 recipes. We also extensively test individual modules with 180 episodes of visuomotor picking, 60 episodes of human motion forecasting, and 46 online user evaluations of the task planner. We show that MOSAIC is able to efficiently collaborate with humans by running the overall system end-to-end with a real human user, completing 68.3% (41/60) collaborative cooking trials of 6 different recipes with a subtask completion rate of 91.6%. Finally, we discuss the limitations of the current system and exciting open challenges in this domain. The project's website is at https://portal-cornell.github.io/MOSAIC/

MOSAIC：一個模塊化的輔助和互動烹飪系統

MOSAIC: A Modular System for Assistive and Interactive Cooking

摘要

Support