3D HAMSTER：通过三维轨迹引导桥接分层视觉语言动作模型中的规划与控制

摘要

分层视觉-语言-动作（VLA）模型通过将高层规划与低层控制解耦，提升了机器人操作的泛化能力。该范式下的近期工作利用视觉语言模型（VLM）预测的2D末端执行器轨迹，作为下游策略的显式引导。然而，当前最先进的低层策略在点云上的3D度量空间中运行，向其输入缺乏深度信息的2D引导时，每个航点必须被赋予其下方场景表面的深度值，从而产生几何畸变的轨迹。我们提出3D HAMSTER，一种通过让规划器直接输出度量可靠的3D轨迹来弥合这一差距的分层框架。我们为VLM配备了专用的深度编码器和密集深度重建目标，用于预测3D航点序列，这些序列被直接集成到基于点云的低层策略中。在3D轨迹预测、仿真和真实世界操作任务中，3D HAMSTER始终优于专有VLM和基于2D引导的基线方法，尤其在出现外观变化偏移以及未见过的语言、空间和视觉条件时性能提升最为显著。项目页面可访问 https://davian-robotics.github.io/3D_HAMSTER/。

English

Hierarchical Vision-Language-Action (VLA) models decouple high-level planning from low-level control to improve generalization in robot manipulation. Recent work in this paradigm uses 2D end-effector trajectories predicted by a Vision-Language Model (VLM) as explicit guidance for a downstream policy. However, state-of-the-art low-level policies operate in 3D metric space on point clouds, and feeding them 2D guidance that lacks depth forces each waypoint to be assigned the depth of whatever scene surface lies beneath it, producing geometrically distorted trajectories. We propose 3D HAMSTER, a hierarchical framework that closes this gap by having the planner directly output metrically reliable 3D trajectories. We augment a VLM with a dedicated depth encoder and a dense depth reconstruction objective to predict 3D waypoint sequences, which are directly integrated into a pointcloudbased low-level policy. Across 3D trajectory prediction, simulation, and real-world manipulation, 3D HAMSTER consistently outperforms proprietary VLMs and 2D-guided baselines, with the largest gains under appearance-altering shifts and unseen language, spatial, and visual conditions. The project page is available at https://davian-robotics.github.io/3D_HAMSTER/.