RxBrain：具身认知基础模型——融合语言-视觉联合推理与想象

摘要

具身认知要求智能体将高层任务推理与待实现的物理状态相连接。本文提出Hy-Embodied-RxBrain，一个具备语言-视觉联合推理与想象能力的具身认知基础模型。不同于侧重场景理解与文本决策的视觉语言模型，也不同于主要预测未来视觉状态的生成式世界模型，RxBrain通过单一规划序列表征具身规划，其中语言与视觉想象发挥互补作用：语言提供规划的抽象结构，包括任务分解、规划基元、约束条件、时序顺序与决策逻辑；而视觉想象则通过世界状态预测与联合子目标规划，将这一结构具体化，将每个规划步骤与中间及最终物理状态相关联。RxBrain采用统一的多模态混合Transformer架构，支持在一个模型中完成语言、图像和视频的理解与生成。为训练该能力，我们构建了一条自动流水线，通过将具身视频分解为规划步骤并与视觉状态转换对齐，将其转化为文本-视觉联合规划监督信号。我们进一步引入RxBrain-Bench评估基准，检验模型是否能够通过联合文本与视觉组件来表征具身规划，而非仅依赖单独的理解或生成能力。实验表明，RxBrain保留了具身理解与生成能力，并能生成耦合文本推理、世界状态预测与联合子目标规划的规划方案。我们还将RxBrain扩展至连续机器人动作生成，该模型在无需大规模动作数据预训练的情况下展现了有前景的实际机器人性能。这些结果为基础模型迈向具身认知提供了初步探索。

English

Embodied cognition requires agents to connect high-level task reasoning with the physical states to be achieved. We introduce Hy-Embodied-RxBrain, an embodied cognition foundation model with joint language-visual reasoning and imagination. Unlike vision-language models that emphasize scene understanding and textual decision making, or generative world models that mainly predict future visual states, RxBrain represents embodied plans in a single planning sequence where language and visual imagination play complementary roles. Language provides the abstract structure of a plan, including task decomposition, planning primitives, constraints, temporal order, and decision logic, while visual imagination grounds this structure through world state prediction and joint subgoal planning, associating each planning step with intermediate and final physical states. RxBrain adopts a unified multimodal Mixture-of-Transformers architecture that supports language, image, and video understanding and generation within one model. To train this capability, we build an automatic pipeline that converts embodied videos into joint text-visual planning supervision by decomposing videos into planning steps and aligning them with visual state transitions. We further introduce RxBrain-Bench to evaluate whether models can represent embodied plans through joint textual and visual components rather than separate understanding or generation. Experiments show that RxBrain maintains embodied understanding and generation abilities, and produces plans with coupled textual reasoning, world state prediction, and joint subgoal planning. We also extend RxBrain to continuous robot action generation, where it shows promising real-robot performance without large-scale action-data pretraining. These results provide an initial step toward foundation models for embodied cognition.