WorldSimBench：走向视频生成模型作为世界模拟器

WorldSimBench: Towards Video Generation Models as World Simulators

October 23, 2024

作者: Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, Zhenfei Yin, Xihui Liu, Lu Sheng, Jing Shao, Lei Bai, Wanli Ouyang, Ruimao Zhang

cs.AI

摘要

最近预测模型的进展展示了在预测物体和场景未来状态方面的卓越能力。然而，基于固有特征的分类不足仍然阻碍了预测模型发展的进展。此外，现有基准无法有效评估具有更高能力、高度具象化的预测模型的具象化视角。在这项工作中，我们将预测模型的功能性分类为层次结构，并通过提出一个名为WorldSimBench的双重评估框架，迈出了评估世界模拟器的第一步。WorldSimBench包括显式感知评估和隐式操纵评估，涵盖了从视觉角度的人类偏好评估和具象化任务中的动作级评估，涵盖了三个具象化场景：开放式具象化环境、自主驾驶和机器人操作。在显式感知评估中，我们引入了HF-具象化数据集，这是一个基于细粒度人类反馈的视频评估数据集，我们用它来训练一个与人类感知一致并明确评估世界模拟器视觉保真度的人类偏好评估器。在隐式操纵评估中，我们通过评估世界模拟器的视频-动作一致性来评估其在动态环境中生成的情境感知视频是否能够准确转化为正确的控制信号。我们全面的评估提供了关键见解，可以推动视频生成模型的进一步创新，将世界模拟器定位为具象化人工智能的重要进展。

English

Recent advancements in predictive models have demonstrated exceptional capabilities in predicting the future state of objects and scenes. However, the lack of categorization based on inherent characteristics continues to hinder the progress of predictive model development. Additionally, existing benchmarks are unable to effectively evaluate higher-capability, highly embodied predictive models from an embodied perspective. In this work, we classify the functionalities of predictive models into a hierarchy and take the first step in evaluating World Simulators by proposing a dual evaluation framework called WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, encompassing human preference assessments from the visual perspective and action-level evaluations in embodied tasks, covering three representative embodied scenarios: Open-Ended Embodied Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment dataset based on fine-grained human feedback, which we use to train a Human Preference Evaluator that aligns with human perception and explicitly assesses the visual fidelity of World Simulators. In the Implicit Manipulative Evaluation, we assess the video-action consistency of World Simulators by evaluating whether the generated situation-aware video can be accurately translated into the correct control signals in dynamic environments. Our comprehensive evaluation offers key insights that can drive further innovation in video generation models, positioning World Simulators as a pivotal advancement toward embodied artificial intelligence.