RynnWorld-4D：面向机器人操作的4D具身世界模型

摘要

开放世界中的机器人操作不仅需要识别场景的外观，还需要预测其三维结构在交互作用下的运动方式。我们认为，同步的RGB、深度和光流（即RGB-DF）提供了一种物理上具有可解释性的表征，能够捕捉场景底层的四维动态。与二维像素视频相比，这种多模态协同将视觉外观、几何结构与时序运动对齐，创建了一个更接近机器人系统所需低层级末端执行器动作的表征空间，从而缩小了世界预测与策略学习之间的鸿沟。基于这一洞察，我们提出了RynnWorld-4D，这是一个生成模型，能够从单张RGB-D图像和一条语言指令出发，在统一的扩散过程中联合生成未来的RGB帧、深度图和光流。该四维世界模型采用三分支架构，融合了跨模态注意力与逐帧三维旋转位置编码（RoPE），确保外观、几何和运动保持一致演化。为了提供大规模训练数据，我们整理了Rynn4DDataset 1.0，这是一个包含超过2.544亿帧的大规模数据集，覆盖自我中心视角的人类和机器人操作视频，并带有高质量的深度和光流伪标签。我们进一步提出了RynnWorld-4D-Policy，这是一个逆动力学头部，能够在前向传播中一次性消耗RynnWorld-4D的内部四维表征，绕过昂贵的多步去噪过程，以闭环方式输出机器人动作。实验表明，RynnWorld-4D能够产生时空一致的四维预测，而RynnWorld-4D-Policy在真实世界灵巧双臂操作任务中达到了最先进的性能，特别是在需要空间精度和时间协调的任务中表现卓越。

English

Robotic manipulation in the open world requires not only recognizing what a scene looks like, but also anticipating how its 3D structure moves under interaction. We argue that synchronized RGB, depth, and optical flow, namely RGB-DF, provide a physically grounded representation that captures the underlying 4D dynamics of a scene. Compared to 2D pixel videos, this multi-modal synergy aligns visual appearance with geometric structure and temporal motion, creating a representation space significantly closer to the low-level end-effector actions demanded by robotic systems, thereby narrowing the gap between world prediction and policy learning. Building on this insight, we introduce RynnWorld-4D, a generative model that co-produces future RGB frames, depth maps, and optical flow from a single RGB-D image and a language instruction within one unified diffusion process. This 4D world model features a tri-branch architecture that integrates cross-modal attention with frame-wise 3D RoPE, ensuring that appearance, geometry, and motion evolve consistently. To supply training data at scale, we curate Rynn4DDataset 1.0, a massive dataset of over 254.4 million frames across egocentric human and robotic manipulation videos with high-quality pseudo-labels for depth and optical flow. We further propose RynnWorld-4D-Policy, an inverse dynamics head that consumes the internal 4D representations of RynnWorld-4D in a single forward pass, bypassing expensive multi-step denoising, to output robot actions in a closed-loop manner. Experiments show that RynnWorld-4D produces temporally and spatially coherent 4D predictions, and that RynnWorld-4D-Policy achieves state-of-the-art performance on real-world dexterous bimanual manipulation tasks, particularly excelling in tasks demanding spatial precision and temporal coordination.