基於視覺的人形機器人靈巧操作之模擬到現實強化學習

Sim-to-Real Reinforcement Learning for Vision-Based Dexterous Manipulation on Humanoids

February 27, 2025

作者: Toru Lin, Kartik Sachdev, Linxi Fan, Jitendra Malik, Yuke Zhu

cs.AI

摘要

強化學習在多樣化的問題領域中已展現出實現人類甚至超人類水準能力的潛力，但在靈巧的機器人操作方面，其成功仍顯有限。本研究探討了將強化學習應用於解決人形機器人上一系列接觸豐富的操作任務時所面臨的關鍵挑戰。我們引入新穎技術來克服這些已識別的挑戰，並通過實證進行驗證。我們的主要貢獻包括：一個自動化的實物到模擬調校模組，使模擬環境更接近真實世界；一個通用的獎勵設計方案，簡化了長期接觸豐富操作任務的獎勵工程；一個分而治之的蒸餾過程，在保持模擬到真實性能的同時，提高了難探索問題的樣本效率；以及稀疏與密集物體表示的混合，以彌合模擬到真實的感知差距。我們在三項人形靈巧操作任務上展示了令人鼓舞的結果，並對每項技術進行了消融研究。我們的工作提出了一種成功的學習人形靈巧操作的方法，利用模擬到真實的強化學習，實現了強大的泛化能力和高性能，而無需人類示範。

English

Reinforcement learning has delivered promising results in achieving human- or even superhuman-level capabilities across diverse problem domains, but success in dexterous robot manipulation remains limited. This work investigates the key challenges in applying reinforcement learning to solve a collection of contact-rich manipulation tasks on a humanoid embodiment. We introduce novel techniques to overcome the identified challenges with empirical validation. Our main contributions include an automated real-to-sim tuning module that brings the simulated environment closer to the real world, a generalized reward design scheme that simplifies reward engineering for long-horizon contact-rich manipulation tasks, a divide-and-conquer distillation process that improves the sample efficiency of hard-exploration problems while maintaining sim-to-real performance, and a mixture of sparse and dense object representations to bridge the sim-to-real perception gap. We show promising results on three humanoid dexterous manipulation tasks, with ablation studies on each technique. Our work presents a successful approach to learning humanoid dexterous manipulation using sim-to-real reinforcement learning, achieving robust generalization and high performance without the need for human demonstration.