HumanCLAW: 视觉语言模型能否通过身体行动？

摘要

评估视觉-语言模型（VLM）能否通过物理身体执行动作是具有挑战性的。动作的结果将VLM的决策与电机控制耦合在一起。当任务失败时，很难判断是VLM做出了错误选择，还是电机控制器未能成功执行（例如，因失去平衡而摔倒）。在这项工作中，我们提出了HumanCLAW——一个将动作决策与底层执行解耦的评估框架。在每一步中，一个被约束的现成VLM发出一个原子技能命令，该命令被转换为亚秒级的连续全身运动，并伴有真实的物理后果（包括重力和碰撞）。因此，身体可以在物理世界中自由行动，而执行侧的干扰（如平衡和电机误差）则被分离出去。剩下可测量的是模型的动作智能：其在每个瞬间选择身体下一步应执行什么。基于这一框架，我们构建了HumanCLAW基准测试：包含1,218个长周期、第一人称视角的“发现-导航-交互”任务片段，覆盖41个室内场景。我们测试了九个最先进的VLM，发现没有一个能解决该基准测试；最佳模型的成功率仅为16.8%。识别目标并非瓶颈所在。当前VLM所缺乏的是具身自我意识：它们会丢失自身身体的位置信息，无法判断身体在哪里、是否到达目标，或是否碰到了障碍物。

English

Evaluating whether a vision-language model (VLM) can act through a physical body is challenging. The outcome of an action couples the VLM's decision with motor control. When a task fails, it is hard to tell whether the VLM made a bad choice or the motor controller simply failed to execute it, e.g., losing balance and falling. In this work, we introduce HumanCLAW, an evaluation framework that decouples action decision-making from low-level execution. At every step, a harnessed, off-the-shelf VLM issues an atomic skill command, and the command is translated into a sub-second chunk of continuous full-body motion with real physical consequences, including gravity and collisions. The body can therefore act freely in the physical world, while execution-side disturbances, balance and motor errors, are factored out. What remains measurable is the model's action intelligence: its moment-to-moment choice of what the body should execute next. Based on this framework, we build HumanCLAW-Bench: 1,218 long-horizon, egocentric find-navigate-interact episodes across 41 indoor scenes. We test nine state-of-the-art VLMs and find that none solves the benchmark; the best model reaches only a 16.8% success rate. Recognizing the target is not the bottleneck. What current VLMs lack is embodied self-awareness: they lose track of their own body, failing to tell where it is, whether it has reached the goal, or whether it has hit an obstacle.