ROS-LLM：一個具有任務反饋和結構化推理的具體AI的ROS框架

ROS-LLM: A ROS framework for embodied AI with task feedback and structured reasoning

June 28, 2024

作者: Christopher E. Mower, Yuhui Wan, Hongzhan Yu, Antoine Grosnit, Jonas Gonzalez-Billandon, Matthieu Zimmer, Jinlong Wang, Xinyu Zhang, Yao Zhao, Anbang Zhai, Puze Liu, Davide Tateo, Cesar Cadena, Marco Hutter, Jan Peters, Guangjian Tian, Yuzheng Zhuang, Kun Shao, Xingyue Quan, Jianye Hao, Jun Wang, Haitham Bou-Ammar

cs.AI

摘要

我們提出了一個框架，用於讓非專家通過自然語言提示和來自機器人操作系統（ROS）的上下文信息進行直觀的機器人編程。我們的系統集成了大型語言模型（LLMs），使非專家能夠通過聊天界面向系統表達任務需求。該框架的關鍵特點包括：將ROS與連接到眾多開源和商業LLMs的人工智能代理整合，從LLM輸出中自動提取行為並執行ROS動作/服務，支持三種行為模式（序列、行為樹、狀態機），模仿學習以將新的機器人動作添加到可能動作庫中，以及通過人類和環境反饋實現LLM反思。廣泛的實驗驗證了該框架，在各種場景中展示了其穩健性、可擴展性和多功能性，包括長期任務、桌面重新排列和遠程監督控制。為了促進我們框架的應用並支持我們結果的重現，我們已將我們的代碼開源。您可以在以下網址訪問：https://github.com/huawei-noah/HEBO/tree/master/ROSLLM。

English

We present a framework for intuitive robot programming by non-experts, leveraging natural language prompts and contextual information from the Robot Operating System (ROS). Our system integrates large language models (LLMs), enabling non-experts to articulate task requirements to the system through a chat interface. Key features of the framework include: integration of ROS with an AI agent connected to a plethora of open-source and commercial LLMs, automatic extraction of a behavior from the LLM output and execution of ROS actions/services, support for three behavior modes (sequence, behavior tree, state machine), imitation learning for adding new robot actions to the library of possible actions, and LLM reflection via human and environment feedback. Extensive experiments validate the framework, showcasing robustness, scalability, and versatility in diverse scenarios, including long-horizon tasks, tabletop rearrangements, and remote supervisory control. To facilitate the adoption of our framework and support the reproduction of our results, we have made our code open-source. You can access it at: https://github.com/huawei-noah/HEBO/tree/master/ROSLLM.