DexTrack：实现从人类参考中实现灵巧操作的通用神经跟踪控制

DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human References

February 13, 2025

作者: Xueyi Liu, Jianibieke Adalibieke, Qianwei Han, Yuzhe Qin, Li Yi

cs.AI

摘要

我们致力于应对从人类参考中开发出适用于灵巧操作的通用神经跟踪控制器的挑战。该控制器旨在管理一个灵巧机器人手，以根据人体与物体之间的运动学相互作用定义的各种目的来操纵不同的物体。开发这样的控制器受到灵巧操作复杂的接触动力学和对适应性、通用性和稳健性的需求的影响。当前的强化学习和轨迹优化方法通常由于依赖于特定任务奖励或精确系统模型而表现不佳。我们提出了一种方法，通过筛选大规模成功的机器人跟踪演示，包括人类参考和机器人动作的配对，来训练一个神经控制器。利用数据飞轮，我们迭代地提升控制器的性能，以及成功跟踪演示的数量和质量。我们利用可用的跟踪演示，并精心整合强化学习和模仿学习，以提高控制器在动态环境中的性能。同时，为了获得高质量的跟踪演示，我们通过利用学习的跟踪控制器在同伦优化方法中优化每条轨迹的跟踪。同伦优化，模拟思维链，有助于解决具有挑战性的轨迹跟踪问题，增加演示的多样性。我们展示了通过训练一个通用神经控制器并在模拟和真实世界中评估其性能的成功。与主流基线相比，我们的方法成功率提高了超过10%。项目网站上提供了带有动画结果的链接：https://meowuu7.github.io/DexTrack/。

English

We address the challenge of developing a generalizable neural tracking controller for dexterous manipulation from human references. This controller aims to manage a dexterous robot hand to manipulate diverse objects for various purposes defined by kinematic human-object interactions. Developing such a controller is complicated by the intricate contact dynamics of dexterous manipulation and the need for adaptivity, generalizability, and robustness. Current reinforcement learning and trajectory optimization methods often fall short due to their dependence on task-specific rewards or precise system models. We introduce an approach that curates large-scale successful robot tracking demonstrations, comprising pairs of human references and robot actions, to train a neural controller. Utilizing a data flywheel, we iteratively enhance the controller's performance, as well as the number and quality of successful tracking demonstrations. We exploit available tracking demonstrations and carefully integrate reinforcement learning and imitation learning to boost the controller's performance in dynamic environments. At the same time, to obtain high-quality tracking demonstrations, we individually optimize per-trajectory tracking by leveraging the learned tracking controller in a homotopy optimization method. The homotopy optimization, mimicking chain-of-thought, aids in solving challenging trajectory tracking problems to increase demonstration diversity. We showcase our success by training a generalizable neural controller and evaluating it in both simulation and real world. Our method achieves over a 10% improvement in success rates compared to leading baselines. The project website with animated results is available at https://meowuu7.github.io/DexTrack/.