FastTD3：面向人形机器人控制的简约、高效且强大的强化学习算法

FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control

May 28, 2025

作者: Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Zhao-Heng Yin, Pieter Abbeel

cs.AI

摘要

强化学习（RL）在机器人领域推动了显著进展，但其复杂性和漫长的训练时间仍是主要瓶颈。本报告介绍了FastTD3，一种简单、快速且高效的RL算法，它显著加快了在HumanoidBench、IsaacLab和MuJoCo Playground等流行套件中的人形机器人训练速度。我们的方法极为简洁：通过并行模拟、大批量更新、分布式的评价器以及精心调优的超参数，训练一个离策略的TD3智能体。FastTD3在单个A100 GPU上不到3小时即可解决一系列HumanoidBench任务，同时保持训练过程中的稳定性。我们还提供了FastTD3的轻量级、易于使用的实现，以加速机器人领域的RL研究。

English

Reinforcement learning (RL) has driven significant progress in robotics, but its complexity and long training times remain major bottlenecks. In this report, we introduce FastTD3, a simple, fast, and capable RL algorithm that significantly speeds up training for humanoid robots in popular suites such as HumanoidBench, IsaacLab, and MuJoCo Playground. Our recipe is remarkably simple: we train an off-policy TD3 agent with several modifications -- parallel simulation, large-batch updates, a distributional critic, and carefully tuned hyperparameters. FastTD3 solves a range of HumanoidBench tasks in under 3 hours on a single A100 GPU, while remaining stable during training. We also provide a lightweight and easy-to-use implementation of FastTD3 to accelerate RL research in robotics.