RT-Sketch：從手繪草圖進行目標條件下的模仿學習

摘要

在目標條件模仿學習（IL）中，自然語言和圖像通常被用作目標表示。然而，自然語言可能存在歧義，而圖像可能過度具體。在這項工作中，我們提出手繪草圖作為視覺模仿學習中目標規定的一種形式。草圖易於用戶即時提供，類似於語言，但與圖像一樣，它們也可以幫助下游策略具有空間感知能力，甚至超越圖像以區分任務相關與無關的對象。我們提出了RT-Sketch，這是一個以目標為條件的操作策略，接受所需場景的手繪草圖作為輸入，並輸出動作。我們在一組配對軌跡和相應的合成目標草圖數據集上訓練RT-Sketch。我們在一個可移動的櫃檯上評估了這種方法在涉及桌面物體重新排列的六種操作技能上的表現。實驗結果顯示，在直接設置中，RT-Sketch能夠達到與圖像或語言條件的代理相似的水平，同時在語言目標存在歧義或視覺干擾物存在時實現更大的韌性。此外，我們展示了RT-Sketch具有解釋和執行不同特定程度的草圖的能力，從最小的線條草圖到詳細的彩色草圖。有關補充材料和視頻，請參閱我們的網站：http://rt-sketch.github.io。

English

Natural language and images are commonly used as goal representations in goal-conditioned imitation learning (IL). However, natural language can be ambiguous and images can be over-specified. In this work, we propose hand-drawn sketches as a modality for goal specification in visual imitation learning. Sketches are easy for users to provide on the fly like language, but similar to images they can also help a downstream policy to be spatially-aware and even go beyond images to disambiguate task-relevant from task-irrelevant objects. We present RT-Sketch, a goal-conditioned policy for manipulation that takes a hand-drawn sketch of the desired scene as input, and outputs actions. We train RT-Sketch on a dataset of paired trajectories and corresponding synthetically generated goal sketches. We evaluate this approach on six manipulation skills involving tabletop object rearrangements on an articulated countertop. Experimentally we find that RT-Sketch is able to perform on a similar level to image or language-conditioned agents in straightforward settings, while achieving greater robustness when language goals are ambiguous or visual distractors are present. Additionally, we show that RT-Sketch has the capacity to interpret and act upon sketches with varied levels of specificity, ranging from minimal line drawings to detailed, colored drawings. For supplementary material and videos, please refer to our website: http://rt-sketch.github.io.

RT-Sketch：從手繪草圖進行目標條件下的模仿學習

RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches

摘要

Support