VibeWorlding：多模态智能体能否端到端地构建3D开放世界？

摘要

根据用户查询构建交互式3D开放世界具有重要意义。然而，现有方法主要是在理想化、简单的查询上进行评估，这使得难以系统性地分析和比较多模态智能体如何理解用户意图、使用3D工具以及基于文本和视觉3D世界信息进行推理。为此，我们提出了VibeWorlding，一个用于对vibe worlding智能体进行基准测试和训练的统一框架。vibe worlding智能体是一种多模态智能体，能够在多轮智能体-环境交互过程中自主推断用户意图、规划场景布局、调用3D工具，并对多模态反馈进行反思。为实现这一目标，我们首先构建了VWE-BENCH基准，包含2,616个高质量3D资产、323个人工标注的种子3D世界以及6,828条反向合成的多模态用户查询，这些查询分为具有真值的已验证查询和带有精心设计量规的未验证查询。此外，我们开发了VibeWorlding-Gym，一个联合多模态强化学习后训练框架，它整合了（1）一个沙盒环境，将资产检索、编辑和图像渲染统一为MCP工具，以及（2）一个基于量规的验证器，结合物理可行性和意图实现验证，既支持公平的模型评估，也支持可扩展的多模态强化学习奖励服务。我们的实验表明，当前前沿的多模态大语言模型（MLLMs）远未解决vibe worlding智能体任务，即使是GPT-5.5和Qwen3.8-Max的成功率也低于60%，并将瓶颈归结为精确的3D世界编辑。我们进一步发现，强化学习训练可以缓解这一弱点，并使开源多模态大语言模型甚至超越闭源前沿模型：我们的VibeWorlder-8B与前沿MLLMs相当，而我们的旗舰模型VibeWorlder-30B-A3B在所有评估模型中取得了最佳的整体Pass@1。

English

Constructing an interactive 3D open world from a user query is important. However, existing methods are primarily evaluated on idealized, simple queries, making it difficult to systematically analyze and compare how multimodal agents understand user intent, use 3D tools, and reason over textual and visual 3D world information. To this end, we propose VibeWorlding, a unified framework for benchmarking and training vibe worlding agents: a multimodal agent that can autonomously infer user intent, plan scene layout, invoke 3D tools, and reflect on the multimodal feedback in a multi-turn agent-environment interaction process. To achieve this, we first build VWE-BENCH, a benchmark of 2,616 high-quality 3D assets, 323 human-annotated seed 3D worlds, and 6,828 reverse-synthesized multimodal user queries, split into verified queries with ground-truth and unverified queries with carefully designed rubrics. Moreover, we develop VibeWorlding-Gym, a joint multimodal RL post-training framework that integrates (1) a sandbox environment unifying asset retrieval, editing, and image rendering as MCP tools, and (2) a rubric-based verifier that combines physical feasibility and intent fulfillment verification, supporting both fair model evaluation and scalable multimodal RL reward service. Our experiments show that current frontier MLLMs are far from solving the vibe worlding agent task, with even GPT-5.5 and Qwen3.8-Max reaching below 60% success rate, and trace the bottleneck to precise 3D world editing. We further find that RL training can ease this weakness and enable open-source MLLMs to even surpass closed-source frontiers: our VibeWorlder-8B is comparable to frontier MLLMs, while our flagship VibeWorlder-30B-A3B attains the best overall Pass@1 among all evaluated models.