HERMES：基於多源運動數據的人機交互式學習系統 ——面向移動靈巧操作的實現

HERMES: Human-to-Robot Embodied Learning from Multi-Source Motion Data for Mobile Dexterous Manipulation

August 27, 2025

作者: Zhecheng Yuan, Tianming Wei, Langzhe Gu, Pu Hua, Tianhai Liang, Yuanpei Chen, Huazhe Xu

cs.AI

摘要

利用人類運動數據賦予機器人多功能操作技能，已成為機器人操作領域中一個頗具前景的範式。然而，將多源的人類手部運動轉化為可行的機器人行為仍面臨挑戰，尤其是對於配備多指靈巧手的機器人而言，其複雜的高維動作空間更增加了難度。此外，現有方法往往難以生成能夠適應多樣環境條件的策略。本文介紹了HERMES，一個面向移動雙手機器人靈巧操作的人機學習框架。首先，HERMES構建了一種統一的強化學習方法，能夠無縫地將來自多源的異構人類手部運動轉化為物理上合理的機器人行為。隨後，為縮小仿真與現實的差距，我們設計了一種基於深度圖像的端到端仿真到現實轉移方法，以提升對現實場景的泛化能力。進一步地，為了在變化和非結構化環境中實現自主操作，我們在導航基礎模型中引入了閉環的透視n點（PnP）定位機制，確保視覺目標的精確對齊，有效橋接了自主導航與靈巧操作。大量實驗結果表明，HERMES在多樣化的野外場景中展現出良好的泛化行為，成功完成了多項複雜的移動雙手機器人靈巧操作任務。項目頁面：https://gemcollector.github.io/HERMES/。

English

Leveraging human motion data to impart robots with versatile manipulation skills has emerged as a promising paradigm in robotic manipulation. Nevertheless, translating multi-source human hand motions into feasible robot behaviors remains challenging, particularly for robots equipped with multi-fingered dexterous hands characterized by complex, high-dimensional action spaces. Moreover, existing approaches often struggle to produce policies capable of adapting to diverse environmental conditions. In this paper, we introduce HERMES, a human-to-robot learning framework for mobile bimanual dexterous manipulation. First, HERMES formulates a unified reinforcement learning approach capable of seamlessly transforming heterogeneous human hand motions from multiple sources into physically plausible robotic behaviors. Subsequently, to mitigate the sim2real gap, we devise an end-to-end, depth image-based sim2real transfer method for improved generalization to real-world scenarios. Furthermore, to enable autonomous operation in varied and unstructured environments, we augment the navigation foundation model with a closed-loop Perspective-n-Point (PnP) localization mechanism, ensuring precise alignment of visual goals and effectively bridging autonomous navigation and dexterous manipulation. Extensive experimental results demonstrate that HERMES consistently exhibits generalizable behaviors across diverse, in-the-wild scenarios, successfully performing numerous complex mobile bimanual dexterous manipulation tasks. Project Page:https://gemcollector.github.io/HERMES/.