三维场景生成技术综述

3D Scene Generation: A Survey

May 8, 2025

作者: Beichen Wen, Haozhe Xie, Zhaoxi Chen, Fangzhou Hong, Ziwei Liu

cs.AI

摘要

三维场景生成旨在为沉浸式媒体、机器人技术、自动驾驶以及具身智能等应用合成具有空间结构、语义意义且逼真的环境。早期基于程序规则的方法虽具备可扩展性，但多样性受限。近年来，深度生成模型（如GANs、扩散模型）与三维表示技术（如NeRF、3D高斯）的进步，使得学习真实世界场景分布成为可能，从而提升了生成结果的逼真度、多样性及视角一致性。特别是扩散模型，通过将生成问题重构为图像或视频合成任务，成功架起了三维场景生成与照片级真实感之间的桥梁。本综述系统梳理了当前最先进的方法，将其归纳为四大范式：程序化生成、基于神经网络的3D生成、基于图像的生成以及基于视频的生成。我们深入分析了这些方法的技术基础、权衡取舍及代表性成果，并回顾了常用的数据集、评估协议及下游应用。最后，我们探讨了生成能力、三维表示、数据与标注、评估等方面的关键挑战，并展望了包括更高逼真度、物理感知与交互式生成、以及统一感知-生成模型在内的未来发展方向。本综述不仅梳理了三维场景生成的最新进展，还强调了生成式AI、三维视觉与具身智能交叉领域的潜在研究方向。为追踪最新动态，我们维护了一个持续更新的项目页面：https://github.com/hzxie/Awesome-3D-Scene-Generation。

English

3D scene generation seeks to synthesize spatially structured, semantically meaningful, and photorealistic environments for applications such as immersive media, robotics, autonomous driving, and embodied AI. Early methods based on procedural rules offered scalability but limited diversity. Recent advances in deep generative models (e.g., GANs, diffusion models) and 3D representations (e.g., NeRF, 3D Gaussians) have enabled the learning of real-world scene distributions, improving fidelity, diversity, and view consistency. Recent advances like diffusion models bridge 3D scene synthesis and photorealism by reframing generation as image or video synthesis problems. This survey provides a systematic overview of state-of-the-art approaches, organizing them into four paradigms: procedural generation, neural 3D-based generation, image-based generation, and video-based generation. We analyze their technical foundations, trade-offs, and representative results, and review commonly used datasets, evaluation protocols, and downstream applications. We conclude by discussing key challenges in generation capacity, 3D representation, data and annotations, and evaluation, and outline promising directions including higher fidelity, physics-aware and interactive generation, and unified perception-generation models. This review organizes recent advances in 3D scene generation and highlights promising directions at the intersection of generative AI, 3D vision, and embodied intelligence. To track ongoing developments, we maintain an up-to-date project page: https://github.com/hzxie/Awesome-3D-Scene-Generation.