像机器人一样看：以机器人为中心的点图用于视觉-语言-动作模型

摘要

视觉-语言-动作（VLA）模型通过视觉观测和语言指令预测机器人动作。这些动作在机器人自身的3D坐标系中定义，但大多数VLA模型从相机视角观察场景，导致场景观测与动作定义之间存在帧失配。在固定视角下，该失配问题较轻微——策略可记忆单一观测到动作的映射关系；但当大规模数据集汇集不同相机设置下的示教数据、且策略需要跨视角泛化该映射时，问题会显著加剧。我们提出以机器人为中心的点图（pointmaps）来解决这一失配问题：该类图像的像素存储场景点在机器人坐标系中的3D坐标。点图在保留预训练2D VLA所需的密集H×W网格结构的同时，提供了机器人框架下的3D几何信息，因此能以极小的架构改动集成至现有VLA模型中。在RoboCasa数据集上，点图同时改进了pi0.5和SmolVLA模型，并优于代表性的相机视角方法和3D感知基线。在真实机器人实验中，当相机被移至训练中未出现的放置位置时，基于点图的策略相比纯RGB策略的优势进一步扩大。

English

Vision-language-action (VLA) models predict robot actions from visual observations and language instructions. These actions are defined in the robot's own 3D coordinate frame, yet most VLAs observe the scene in the camera frame, creating a frame mismatch between where the scene is observed and where actions are defined. The mismatch is benign under a fixed viewpoint, where the policy can memorize a single observation-to-action mapping, but grows harder as large-scale datasets aggregate demonstrations across diverse camera setups and the policy must generalize this mapping across viewpoints. We address this mismatch with robot-centric pointmaps, images whose pixels store the 3D coordinates of scene points in the robot frame. Pointmaps provide robot-frame 3D geometry while preserving the dense H x W grid expected by pretrained 2D VLAs, so they integrate into existing VLAs with minimal architectural change. On RoboCasa, pointmaps improve both pi0.5 and SmolVLA and outperform representative camera-viewpoint and 3D-aware baselines. In real-robot experiments, their advantage over an RGB-only policy widens when the camera is moved to a placement unseen during training.