RT-Sketch：从手绘草图进行目标条件下的模仿学习

RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches

March 5, 2024

作者: Priya Sundaresan, Quan Vuong, Jiayuan Gu, Peng Xu, Ted Xiao, Sean Kirmani, Tianhe Yu, Michael Stark, Ajinkya Jain, Karol Hausman, Dorsa Sadigh, Jeannette Bohg, Stefan Schaal

cs.AI

摘要

在目标条件的模仿学习（IL）中，自然语言和图像通常被用作目标表示。然而，自然语言可能存在歧义，图像可能过度详细。在这项工作中，我们提出手绘草图作为视觉模仿学习中目标规定的一种形式。草图易于用户即时提供，类似于语言，但与图像类似，它们还可以帮助下游策略具有空间感知能力，甚至超越图像以区分任务相关和任务无关的对象。我们提出了RT-Sketch，这是一个用于操作的目标条件策略，它以所需场景的手绘草图作为输入，并输出动作。我们在一组配对轨迹和相应的合成生成目标草图的数据集上对RT-Sketch进行训练。我们在一个关节式台面上涉及桌面物体重新排列的六种操作技能上评估了这种方法。实验结果表明，在简单设置中，RT-Sketch能够表现出与图像或语言条件代理相似的水平，同时在语言目标含糊不清或存在视觉干扰时具有更强的鲁棒性。此外，我们展示了RT-Sketch具有解释和执行具有不同特定级别的草图的能力，从简单的线条草图到详细的彩色草图。有关补充材料和视频，请访问我们的网站：http://rt-sketch.github.io。

English

Natural language and images are commonly used as goal representations in goal-conditioned imitation learning (IL). However, natural language can be ambiguous and images can be over-specified. In this work, we propose hand-drawn sketches as a modality for goal specification in visual imitation learning. Sketches are easy for users to provide on the fly like language, but similar to images they can also help a downstream policy to be spatially-aware and even go beyond images to disambiguate task-relevant from task-irrelevant objects. We present RT-Sketch, a goal-conditioned policy for manipulation that takes a hand-drawn sketch of the desired scene as input, and outputs actions. We train RT-Sketch on a dataset of paired trajectories and corresponding synthetically generated goal sketches. We evaluate this approach on six manipulation skills involving tabletop object rearrangements on an articulated countertop. Experimentally we find that RT-Sketch is able to perform on a similar level to image or language-conditioned agents in straightforward settings, while achieving greater robustness when language goals are ambiguous or visual distractors are present. Additionally, we show that RT-Sketch has the capacity to interpret and act upon sketches with varied levels of specificity, ranging from minimal line drawings to detailed, colored drawings. For supplementary material and videos, please refer to our website: http://rt-sketch.github.io.