基于物体中心的残差强化学习用于零样本仿真到现实VLA增强

摘要

视觉-语言-动作（VLA）模型能够泛化到多种操作任务，但其基于模仿学习的策略在执行精确物理交互时，由于累积执行误差而显得脆弱；那么，一种完全在仿真环境中训练的强化学习策略能否零样本提升现实世界中VLA模型的鲁棒性？残差强化学习通过在冻结的VLA模型之上学习修正策略，提供了天然的框架，但现有方法面临根本性的仿真到现实困境：特权状态方法需要经过有损的知识蒸馏才能部署；基于图像的方法受制于视觉域差异；而现实世界中的强化学习成本高昂且存在安全隐患。我们提出了一种以物体为中心的残差强化学习框架，通过物体姿态来优化VLA动作，构建了在仿真与现实之间保持一致的紧凑观测空间。为了对齐两个领域，我们还在仿真中回放相同的遥操作演示，训练现实世界VLA的仿真对应版本。残差强化学习策略仅在仿真环境中通过姿态噪声注入和随机丢弃进行训练，并零样本迁移到真实机器人。在Franka Research 3（FR3）真实机器人上进行的五项操作任务中，我们的方法将成功率从42%零样本提升至76%，且改进后的 rollout 可进一步用于重新训练基础VLA模型，无需额外遥操作即可实现自我提升。项目页面：https://www.microsoft.com/en-us/research/articles/object-centric-residual-rl/

English

Vision-Language-Action (VLA) models can generalize across diverse manipulation tasks, but their imitation-learning-based policies remain brittle in precise physical interactions due to compounding execution errors; Can a reinforcement learning policy trained purely in simulation improve the robustness of real-world VLAs zero-shot? Residual RL, which learns a corrective policy on top of a frozen VLA, offers a natural framework, but existing approaches face a fundamental sim-to-real dilemma: privileged-state methods require lossy distillation for deployment; image-based methods suffer from the visual domain gap; and real-world RL is costly and unsafe. We propose an object-centric residual RL framework that refines VLA actions using object poses, enabling a compact observation space that transfers consistently between simulation and reality. To align the two domains, we additionally replay the same teleoperation demonstrations in simulation to train a sim counterpart of the real-world VLA. The residual RL policy is trained only in simulation with pose noise injection and dropout, and transfers zero-shot to the real robot. Across five manipulation tasks on a real Franka Research 3 (FR3) robot, our method improves the success rate from 42% to 76% zero-shot, and the improved rollouts can be further reused to retrain the base VLA for self-improvement without additional teleoperation. Project page: https://www.microsoft.com/en-us/research/articles/object-centric-residual-rl/