用于分层规划的组合式基础模型

摘要

为了在具有长期目标的新环境中做出有效决策，跨越空间和时间尺度进行分层推理至关重要。这包括规划抽象的子目标序列，通过视觉推理了解底层计划，并根据设计的计划通过视觉-运动控制执行动作。我们提出了用于分层规划的组合基础模型（HiP），这是一个基础模型，利用分别在语言、视觉和行动数据上训练的多个专家基础模型共同解决长期任务。我们使用大型语言模型构建在环境中扎根的符号计划，通过大型视频扩散模型。生成的视频计划然后通过一个从生成的视频中推断动作的逆动力学模型，扎根到视觉-运动控制。为了在这种层次结构内实现有效推理，我们通过迭代细化强化模型之间的一致性。我们通过三个不同的长期桌面操作任务展示了我们方法的有效性和适应性。

English

To make effective decisions in novel environments with long-horizon goals, it is crucial to engage in hierarchical reasoning across spatial and temporal scales. This entails planning abstract subgoal sequences, visually reasoning about the underlying plans, and executing actions in accordance with the devised plan through visual-motor control. We propose Compositional Foundation Models for Hierarchical Planning (HiP), a foundation model which leverages multiple expert foundation model trained on language, vision and action data individually jointly together to solve long-horizon tasks. We use a large language model to construct symbolic plans that are grounded in the environment through a large video diffusion model. Generated video plans are then grounded to visual-motor control, through an inverse dynamics model that infers actions from generated videos. To enable effective reasoning within this hierarchy, we enforce consistency between the models via iterative refinement. We illustrate the efficacy and adaptability of our approach in three different long-horizon table-top manipulation tasks.