OmniNWM：全知驾驶导航世界模型

摘要

自动驾驶世界模型需在状态、动作和奖励三个核心维度上高效运作。然而，现有模型通常局限于有限的状态模态、短视频序列、不精确的动作控制以及缺乏奖励意识。本文中，我们提出了OmniNWM，一种全知全景导航世界模型，它在一个统一框架内解决了所有三个维度的问题。在状态方面，OmniNWM联合生成了包含RGB、语义、度量深度和3D占用的全景视频。通过灵活的强制策略，实现了高质量的长时域自回归生成。对于动作，我们引入了一种归一化的全景Plucker射线图表示法，将输入轨迹编码为像素级信号，从而实现对全景视频生成的高度精确且可泛化的控制。关于奖励，我们超越了依赖外部基于图像的模型学习奖励函数的方法，转而利用生成的3D占用直接定义基于规则的密集奖励，以确保驾驶合规性和安全性。大量实验表明，OmniNWM在视频生成、控制精度和长时域稳定性方面均达到了业界领先水平，同时通过基于占用的奖励提供了可靠的闭环评估框架。项目页面详见https://github.com/Arlo0o/OmniNWM。

English

Autonomous driving world models are expected to work effectively across three core dimensions: state, action, and reward. Existing models, however, are typically restricted to limited state modalities, short video sequences, imprecise action control, and a lack of reward awareness. In this paper, we introduce OmniNWM, an omniscient panoramic navigation world model that addresses all three dimensions within a unified framework. For state, OmniNWM jointly generates panoramic videos of RGB, semantics, metric depth, and 3D occupancy. A flexible forcing strategy enables high-quality long-horizon auto-regressive generation. For action, we introduce a normalized panoramic Plucker ray-map representation that encodes input trajectories into pixel-level signals, enabling highly precise and generalizable control over panoramic video generation. Regarding reward, we move beyond learning reward functions with external image-based models: instead, we leverage the generated 3D occupancy to directly define rule-based dense rewards for driving compliance and safety. Extensive experiments demonstrate that OmniNWM achieves state-of-the-art performance in video generation, control accuracy, and long-horizon stability, while providing a reliable closed-loop evaluation framework through occupancy-grounded rewards. Project page is available at https://github.com/Arlo0o/OmniNWM.

OmniNWM：全知驾驶导航世界模型

OmniNWM: Omniscient Driving Navigation World Models

摘要

Support