以物體為中心的殘差強化學習用於零樣本模擬至真實VLA增強

摘要

視覺-語言-動作（VLA）模型能泛化至多樣化的操作任務，但其基於模仿學習的策略在精確物理互動中仍顯脆弱，原因在於執行誤差的累積；純粹在模擬環境中訓練的強化學習策略，能否零樣本提升真實世界VLA的穩健性？殘差強化學習（Residual RL）——在凍結的VLA之上學習校正策略——提供了自然的框架，但現有方法面臨根本的模擬到真實困境：特權狀態方法需經有損蒸餾才能部署；基於影像的方法受視覺域差距所困；而真實世界強化學習既昂貴又不安全。我們提出以物體為中心的殘差強化學習框架，利用物體姿態精煉VLA動作，從而建構出能在模擬與現實間一致轉換的緊湊觀測空間。為對齊兩域，我們另將相同的遙控示範在模擬中重播，以訓練真實世界VLA的模擬對應版本。殘差強化學習策略僅在模擬中訓練，並加入姿態噪聲注入與隨機丟棄，再零樣本轉移至真實機器人。在真實Franka Research 3（FR3）機器人的五項操作任務中，我們的方法將成功率從42%零樣本提升至76%，且改善後的執行軌跡可進一步用於重新訓練基底VLA，以實現自我改進，無需額外遙控示範。專案頁面：https://www.microsoft.com/en-us/research/articles/object-centric-residual-rl/

English

Vision-Language-Action (VLA) models can generalize across diverse manipulation tasks, but their imitation-learning-based policies remain brittle in precise physical interactions due to compounding execution errors; Can a reinforcement learning policy trained purely in simulation improve the robustness of real-world VLAs zero-shot? Residual RL, which learns a corrective policy on top of a frozen VLA, offers a natural framework, but existing approaches face a fundamental sim-to-real dilemma: privileged-state methods require lossy distillation for deployment; image-based methods suffer from the visual domain gap; and real-world RL is costly and unsafe. We propose an object-centric residual RL framework that refines VLA actions using object poses, enabling a compact observation space that transfers consistently between simulation and reality. To align the two domains, we additionally replay the same teleoperation demonstrations in simulation to train a sim counterpart of the real-world VLA. The residual RL policy is trained only in simulation with pose noise injection and dropout, and transfers zero-shot to the real robot. Across five manipulation tasks on a real Franka Research 3 (FR3) robot, our method improves the success rate from 42% to 76% zero-shot, and the improved rollouts can be further reused to retrain the base VLA for self-improvement without additional teleoperation. Project page: https://www.microsoft.com/en-us/research/articles/object-centric-residual-rl/