WorldToken：面向机器人模仿学习的时间优先序列建模

摘要

机器人策略在每个决策步骤都会接收异构观测，然而序列模型在如何随时间组织这些输入上各不相同。我们提出了WorldToken，这是一种时间优先的策略实例化，它将每个策略时间步内的多视图图像、本体感觉和任务条件融合为一个世界token。一个因果时间Transformer对生成的世界token序列进行建模，并由一个扩散动作头生成动作块。在23个RoboCasa任务上，一个除冻结的预训练CLIP文本编码器外从头训练的8530万参数策略，在每项任务使用2,900个生成的演示的情况下，实现了59.45%的平均闭环成功率。对五种数据集规模、五种模型规模和两种训练随机种子的完整析因扫描显示，额外的目标域数据带来一致的增益，而超过中等模型规模后收益递减。在相同检查点的历史截断下，将可见历史减少到一或两个策略时间步会降低全部50个RoboCasa策略的闭环成功率。在RMBench Blocks Ranking上，将可见历史从146秒缩减到8秒会使评估器成功率从95%降至28%，而探索性扩展推演则将参考交换序列维持了超过850秒。这些结果实证了完整WorldToken实例化的可行性，并刻画了其在所测试的实验方案下的数据缩放和时间上下文行为。它们并未证明其相对于其他序列组织的优越性，也未分离出完整实现中哪些组件驱动了观察到的性能。

English

Robot policies receive heterogeneous observations at each decision step, yet sequence models differ in how they organize these inputs over time. We introduce WorldToken, a time-first policy instantiation that fuses multiview images, proprioception, and task conditioning within each policy timestep into one world token. A causal temporal Transformer models the resulting world-token sequence, and a diffusion action head generates action chunks. On 23 RoboCasa tasks, an 85.3M-parameter policy trained from scratch apart from a frozen pretrained CLIP text encoder achieves 59.45% mean closed-loop success using 2,900 generated demonstrations per task. A complete factorial sweep over five dataset sizes, five model sizes, and two training seeds shows consistent gains from additional target-domain data and diminishing returns beyond moderate model size. Under same-checkpoint history truncation, reducing visible history to one or two policy timesteps lowers closed-loop success for all 50 RoboCasa policies. On RMBench Blocks Ranking, reducing visible history from 146 to 8 seconds lowers evaluator success from 95% to 28%, while an exploratory extended rollout sustains the reference swap sequence for over 850 seconds. These results establish the empirical feasibility of the complete WorldToken instantiation and characterize its data-scaling and temporal-context behavior under the tested recipes. They do not establish superiority over alternative sequence organizations or isolate which components of the complete implementation drive the observed performance.