学习从动作到世界建模的可迁移动态先验

摘要

我们研究基于动作条件的世界建模，将其作为学习机器人可迁移动力学先验的可扩展方法。通过预训练一个能够预测动作如何驱动视觉场景演变的模型，所得世界模型能够捕获超越外观级视频生成的可复用交互动力学。具体而言，我们在包含真实动作标注的大规模机器人操作数据上预训练了一个多视角交互式基础扩散世界模型——A2World。我们从两个互补视角验证所习得的动力学先验。首先，将A2World适配为任务或场景专用的真实世界模拟器A2World-sim，其长程 rollout 支持基于模拟器的策略评估和可扩展的假设分析，通过世界模型 rollout 替代真实机器人 rollout。其次，基于相同的预训练权重，将A2World适配为视频-动作联合预测模型A2World-policy，该模型在视觉和指令条件约束下预测动作。在仿真基准测试和真实机器人环境中的实验表明，基于动作条件的 World Model 预训练能够产生具有迁移性的动力学先验，这对以模拟器为中心和以策略为中心的机器人学习均具有积极意义。

English

We study action-conditioned world modeling as a scalable way to learn transferable dynamics priors for robot learning. By pretraining a model to predict how actions drive visual scene evolution, the resulting world model captures reusable interaction dynamics beyond appearance-level video generation. Concretely, we pretrain a multi-view interactive base diffusion world model, A2World, on large-scale robot manipulation data with real action annotations. We validate the learned dynamics priors from two complementary perspectives. First, we adapt A2World into a task- or scene-specialized real-world simulator, A2World-sim, whose long-horizon rollouts support simulator-based policy evaluation and scalable what-if analysis by replacing real-robot rollouts with world model rollouts. Second, starting from the same pretrained weights, we adapt A2World into a video-action joint prediction model, A2World-policy, that predicts actions under visual and instruction conditioning. Experiments across simulation benchmarks and real-robot settings demonstrate that action-conditioned world model pretraining yields transferable dynamics priors that benefit both simulator-centric and policy-centric robot learning.