以機器人視角觀看：用於視覺-語言-行動模型的機器人中心點圖

摘要

視覺-語言-動作（VLA）模型根據視覺觀察與語言指令預測機器人動作。這些動作定義在機器人自身的3D座標系中，但多數VLA模型卻在相機座標系中觀察場景，導致場景觀察與動作定義之間的座標系錯配。在固定視角下，此錯配影響較小，因為策略能記住單一觀察到動作的映射關係；然而當大規模資料集匯集來自不同相機設定的示範資料，且策略必須在不同視角間泛化此映射時，錯配問題便更為棘手。我們提出以機器人為中心的點圖（robot-centric pointmaps）來解決此問題：此類影像的像素儲存場景點在機器人座標系中的3D座標。點圖在提供機器人座標系3D幾何資訊的同時，保留了預訓練2D VLA模型所需的密集H×W網格結構，因此僅需極小架構調整即可整合至現有VLA模型。在RoboCasa上，點圖同時改善了pi0.5與SmolVLA模型，並優於代表性的相機視角與3D感知基準方法。在真實機器人實驗中，當相機被移動至訓練期間未出現的位置時，點圖相較於僅使用RGB的策略優勢更加明顯。

English

Vision-language-action (VLA) models predict robot actions from visual observations and language instructions. These actions are defined in the robot's own 3D coordinate frame, yet most VLAs observe the scene in the camera frame, creating a frame mismatch between where the scene is observed and where actions are defined. The mismatch is benign under a fixed viewpoint, where the policy can memorize a single observation-to-action mapping, but grows harder as large-scale datasets aggregate demonstrations across diverse camera setups and the policy must generalize this mapping across viewpoints. We address this mismatch with robot-centric pointmaps, images whose pixels store the 3D coordinates of scene points in the robot frame. Pointmaps provide robot-frame 3D geometry while preserving the dense H x W grid expected by pretrained 2D VLAs, so they integrate into existing VLAs with minimal architectural change. On RoboCasa, pointmaps improve both pi0.5 and SmolVLA and outperform representative camera-viewpoint and 3D-aware baselines. In real-robot experiments, their advantage over an RGB-only policy widens when the camera is moved to a placement unseen during training.