利用大型语言模型进行创造性机器人工具使用

Creative Robot Tool Use with Large Language Models

October 19, 2023

作者: Mengdi Xu, Peide Huang, Wenhao Yu, Shiqi Liu, Xilun Zhang, Yaru Niu, Tingnan Zhang, Fei Xia, Jie Tan, Ding Zhao

cs.AI

摘要

工具使用是高级智能的标志，既体现在动物行为中，也体现在机器人的能力中。本文研究了赋予机器人创造性使用工具的能力的可行性，这涉及到涉及隐含物理约束和长期规划的任务。利用大型语言模型（LLMs），我们开发了RoboTool，这是一个系统，它接受自然语言指令，并输出控制机器人在模拟和真实环境中的可执行代码。RoboTool包括四个关键组件：（i）“分析器”，用于解释自然语言以识别关键任务相关概念，（ii）“规划器”，根据语言输入和关键概念生成全面的策略，（iii）“计算器”，计算每个技能的参数，以及（iv）“编码器”，将这些计划转换为可执行的Python代码。我们的结果表明，RoboTool不仅可以理解显式或隐式的物理约束和环境因素，还可以展示创造性的工具使用。与依赖显式优化的传统任务和运动规划（TAMP）方法不同，我们基于LLM的系统为复杂的机器人任务提供了更灵活、高效和用户友好的解决方案。通过大量实验，我们验证了RoboTool在处理否则无法实现的需要创造性工具使用的任务方面表现出色，从而拓展了机器人系统的能力。我们的项目页面上提供了演示：https://creative-robotool.github.io/。

English

Tool use is a hallmark of advanced intelligence, exemplified in both animal behavior and robotic capabilities. This paper investigates the feasibility of imbuing robots with the ability to creatively use tools in tasks that involve implicit physical constraints and long-term planning. Leveraging Large Language Models (LLMs), we develop RoboTool, a system that accepts natural language instructions and outputs executable code for controlling robots in both simulated and real-world environments. RoboTool incorporates four pivotal components: (i) an "Analyzer" that interprets natural language to discern key task-related concepts, (ii) a "Planner" that generates comprehensive strategies based on the language input and key concepts, (iii) a "Calculator" that computes parameters for each skill, and (iv) a "Coder" that translates these plans into executable Python code. Our results show that RoboTool can not only comprehend explicit or implicit physical constraints and environmental factors but also demonstrate creative tool use. Unlike traditional Task and Motion Planning (TAMP) methods that rely on explicit optimization, our LLM-based system offers a more flexible, efficient, and user-friendly solution for complex robotics tasks. Through extensive experiments, we validate that RoboTool is proficient in handling tasks that would otherwise be infeasible without the creative use of tools, thereby expanding the capabilities of robotic systems. Demos are available on our project page: https://creative-robotool.github.io/.