UniSkill：透過跨體現技能表徵模仿人類影片

UniSkill: Imitating Human Videos via Cross-Embodiment Skill Representations

May 13, 2025

作者: Hanjung Kim, Jaehyun Kang, Hyolim Kang, Meedeum Cho, Seon Joo Kim, Youngwoon Lee

cs.AI

摘要

模仿是人類的一項基本學習機制，使個體能夠通過觀察和模仿專家來學習新任務。然而，將這種能力應用於機器人卻面臨著重大挑戰，這主要是由於人類與機器人在視覺外觀和物理能力上的本質差異。雖然先前的方法通過使用共享場景和任務的跨實體數據集來彌合這一差距，但大規模收集人類與機器人之間對齊的數據並非易事。在本文中，我們提出了UniSkill，這是一種新穎的框架，它能夠從大規模的跨實體視頻數據中學習到與實體無關的技能表示，而無需任何標籤，從而使得從人類視頻提示中提取的技能能夠有效地轉移到僅基於機器人數據訓練的策略上。我們在模擬和真實環境中的實驗表明，我們的跨實體技能成功地指導機器人選擇適當的動作，即使面對未見過的視頻提示也是如此。項目網站可訪問：https://kimhanjung.github.io/UniSkill。

English

Mimicry is a fundamental learning mechanism in humans, enabling individuals to learn new tasks by observing and imitating experts. However, applying this ability to robots presents significant challenges due to the inherent differences between human and robot embodiments in both their visual appearance and physical capabilities. While previous methods bridge this gap using cross-embodiment datasets with shared scenes and tasks, collecting such aligned data between humans and robots at scale is not trivial. In this paper, we propose UniSkill, a novel framework that learns embodiment-agnostic skill representations from large-scale cross-embodiment video data without any labels, enabling skills extracted from human video prompts to effectively transfer to robot policies trained only on robot data. Our experiments in both simulation and real-world environments show that our cross-embodiment skills successfully guide robots in selecting appropriate actions, even with unseen video prompts. The project website can be found at: https://kimhanjung.github.io/UniSkill.