Sketch2Scene：從使用者隨意草圖自動生成互動式3D遊戲場景

Sketch2Scene: Automatic Generation of Interactive 3D Game Scenes from User's Casual Sketches

August 8, 2024

作者: Yongzhi Xu, Yonhon Ng, Yifu Wang, Inkyu Sa, Yunfei Duan, Yang Li, Pan Ji, Hongdong Li

cs.AI

摘要

3D內容生成是許多電腦圖形應用的核心，包括視頻遊戲、電影製作、虛擬現實和擴增實境等。本文提出了一種基於深度學習的新方法，用於自動生成互動且可玩的3D遊戲場景，全部源自用戶輕鬆的提示，如手繪草圖。基於草圖的輸入提供了一種自然且便捷的方式，在內容創建過程中傳達用戶的設計意圖。為了克服學習中的數據不足挑戰（即缺乏大量3D場景的訓練數據），我們的方法利用預訓練的2D去噪擴散模型生成場景的2D圖像作為概念指導。在這個過程中，我們採用等距投影模式來排除未知的相機姿勢，同時獲取場景佈局。從生成的等距圖像中，我們使用預訓練的圖像理解方法將圖像分割為有意義的部分，如離地物體、樹木和建築物，並提取2D場景佈局。這些分割和佈局隨後被餵入程序化內容生成（PCG）引擎，例如Unity或Unreal等3D視頻遊戲引擎，以創建3D場景。生成的3D場景可以無縫集成到遊戲開發環境中，並且可以立即進行遊玩。大量測試表明，我們的方法可以高效生成質量高且互動性強的3D遊戲場景，其佈局緊隨用戶意圖。

English

3D Content Generation is at the heart of many computer graphics applications, including video gaming, film-making, virtual and augmented reality, etc. This paper proposes a novel deep-learning based approach for automatically generating interactive and playable 3D game scenes, all from the user's casual prompts such as a hand-drawn sketch. Sketch-based input offers a natural, and convenient way to convey the user's design intention in the content creation process. To circumvent the data-deficient challenge in learning (i.e. the lack of large training data of 3D scenes), our method leverages a pre-trained 2D denoising diffusion model to generate a 2D image of the scene as the conceptual guidance. In this process, we adopt the isometric projection mode to factor out unknown camera poses while obtaining the scene layout. From the generated isometric image, we use a pre-trained image understanding method to segment the image into meaningful parts, such as off-ground objects, trees, and buildings, and extract the 2D scene layout. These segments and layouts are subsequently fed into a procedural content generation (PCG) engine, such as a 3D video game engine like Unity or Unreal, to create the 3D scene. The resulting 3D scene can be seamlessly integrated into a game development environment and is readily playable. Extensive tests demonstrate that our method can efficiently generate high-quality and interactive 3D game scenes with layouts that closely follow the user's intention.