HERMES++：迈向统一驾驶世界模型，实现三维场景理解与生成

HERMES++: Toward a Unified Driving World Model for 3D Scene Understanding and Generation

April 30, 2026

作者: Xin Zhou, Dingkang Liang, Xiwu Chen, Feiyang Tan, Dingyuan Zhang, Hengshuang Zhao, Xiang Bai

cs.AI

摘要

驾驶世界模型作为自动驾驶的关键技术，通过模拟环境动态实现场景推演。然而现有方法主要关注未来场景生成，往往忽视全面的三维场景理解。另一方面，尽管大语言模型展现出卓越的推理能力，却无法预测未来的几何演变，导致语义理解与物理模拟之间存在显著鸿沟。为弥合这一差距，我们提出HERMES++——一个将三维场景理解与未来几何预测整合到统一框架的驾驶世界模型。通过协同设计，我们的方法解决了这两类任务的独特需求：首先，采用鸟瞰图表征将多视角空间信息整合为与大语言模型兼容的结构；其次，引入LLM增强的世界查询机制以促进理解分支的知识迁移；第三，设计当前-未来关联模块来桥接时间鸿沟，使几何演变受语义上下文调节；最后，为保障结构完整性，采用联合几何优化策略，将显式几何约束与隐式潜在正则化相结合，使内部表征对齐几何感知先验。在多个基准测试上的广泛实验验证了方法的有效性：HERMES++在未来点云预测和三维场景理解任务中均超越专业模型，展现出强劲性能。模型与代码将在https://github.com/H-EmbodVis/HERMESV2 公开。

English

Driving world models serve as a pivotal technology for autonomous driving by simulating environmental dynamics. However, existing approaches predominantly focus on future scene generation, often overlooking comprehensive 3D scene understanding. Conversely, while Large Language Models (LLMs) demonstrate impressive reasoning capabilities, they lack the capacity to predict future geometric evolution, creating a significant disparity between semantic interpretation and physical simulation. To bridge this gap, we propose HERMES++, a unified driving world model that integrates 3D scene understanding and future geometry prediction within a single framework. Our approach addresses the distinct requirements of these tasks through synergistic designs. First, a BEV representation consolidates multi-view spatial information into a structure compatible with LLMs. Second, we introduce LLM-enhanced world queries to facilitate knowledge transfer from the understanding branch. Third, a Current-to-Future Link is designed to bridge the temporal gap, conditioning geometric evolution on semantic context. Finally, to enforce structural integrity, we employ a Joint Geometric Optimization strategy that integrates explicit geometric constraints with implicit latent regularization to align internal representations with geometry-aware priors. Extensive evaluations on multiple benchmarks validate the effectiveness of our method. HERMES++ achieves strong performance, outperforming specialist approaches in both future point cloud prediction and 3D scene understanding tasks. The model and code will be publicly released at https://github.com/H-EmbodVis/HERMESV2.