Sketch2Scene：从用户随意草图自动生成交互式3D游戏场景

Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches

August 8, 2024

作者: Yongzhi Xu, Yonhon Ng, Yifu Wang, Inkyu Sa, Yunfei Duan, Yang Li, Pan Ji, Hongdong Li

cs.AI

摘要

3D内容生成是许多计算机图形应用的核心，包括视频游戏、电影制作、虚拟和增强现实等。本文提出了一种基于深度学习的新方法，用于自动生成互动和可玩的3D游戏场景，全部源自用户的随意提示，比如手绘草图。基于草图的输入提供了一种自然、便捷的方式，在内容创建过程中传达用户的设计意图。为了克服学习中的数据不足挑战（即缺乏大规模的3D场景训练数据），我们的方法利用预训练的2D去噪扩散模型生成场景的2D图像作为概念指导。在这个过程中，我们采用等距投影模式来排除未知的摄像机姿势，同时获取场景布局。从生成的等距图像中，我们使用预训练的图像理解方法将图像分割为有意义的部分，比如离地物体、树木和建筑物，并提取2D场景布局。这些分段和布局随后被输入到程序内容生成（PCG）引擎中，比如Unity或Unreal这样的3D视频游戏引擎，以创建3D场景。生成的3D场景可以无缝集成到游戏开发环境中，并且可以立即进行游玩。广泛的测试表明，我们的方法能够高效生成质量高且互动性强的3D游戏场景，其布局紧密符合用户意图。

English

3D Content Generation is at the heart of many computer graphics applications, including video gaming, film-making, virtual and augmented reality, etc. This paper proposes a novel deep-learning based approach for automatically generating interactive and playable 3D game scenes, all from the user's casual prompts such as a hand-drawn sketch. Sketch-based input offers a natural, and convenient way to convey the user's design intention in the content creation process. To circumvent the data-deficient challenge in learning (i.e. the lack of large training data of 3D scenes), our method leverages a pre-trained 2D denoising diffusion model to generate a 2D image of the scene as the conceptual guidance. In this process, we adopt the isometric projection mode to factor out unknown camera poses while obtaining the scene layout. From the generated isometric image, we use a pre-trained image understanding method to segment the image into meaningful parts, such as off-ground objects, trees, and buildings, and extract the 2D scene layout. These segments and layouts are subsequently fed into a procedural content generation (PCG) engine, such as a 3D video game engine like Unity or Unreal, to create the 3D scene. The resulting 3D scene can be seamlessly integrated into a game development environment and is readily playable. Extensive tests demonstrate that our method can efficiently generate high-quality and interactive 3D game scenes with layouts that closely follow the user's intention.