LucidDreamer：无需预设领域的3D高斯溅射场景生成

LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes

November 22, 2023

作者: Jaeyoung Chung, Suyoung Lee, Hyeongjin Nam, Jaerin Lee, Kyoung Mu Lee

cs.AI

摘要

随着VR设备与内容的广泛普及，对三维场景生成技术的需求日益增长。然而，现有的三维场景生成模型受限于采用远离真实世界的三维扫描数据集进行训练的策略，通常将目标场景限定在特定领域。为突破这一局限，我们提出LucidDreamer——一种通过充分发挥现有大规模扩散生成模型优势的无领域限制场景生成流程。该流程包含"造梦"与"对齐"两个交替步骤：首先，为从输入生成多视角一致图像，我们将点云作为各视角图像生成的几何指引。具体而言，通过将局部点云投影至目标视角，并利用该投影作为生成模型进行修复绘制的引导。修复后的图像结合预估深度图被提升至三维空间，形成新点云。其次，为将新点云聚合至三维场景，我们提出一种对齐算法，可协调融合新生成的三维场景局部。最终获得的三维场景将作为优化高斯溅射的初始点云。相较于传统三维场景生成方法，LucidDreamer生成的高斯溅射模型具有更高细节度，且对目标场景领域无任何限制。

English

With the widespread usage of VR devices and contents, demands for 3D scene generation techniques become more popular. Existing 3D scene generation models, however, limit the target scene to specific domain, primarily due to their training strategies using 3D scan dataset that is far from the real-world. To address such limitation, we propose LucidDreamer, a domain-free scene generation pipeline by fully leveraging the power of existing large-scale diffusion-based generative model. Our LucidDreamer has two alternate steps: Dreaming and Alignment. First, to generate multi-view consistent images from inputs, we set the point cloud as a geometrical guideline for each image generation. Specifically, we project a portion of point cloud to the desired view and provide the projection as a guidance for inpainting using the generative model. The inpainted images are lifted to 3D space with estimated depth maps, composing a new points. Second, to aggregate the new points into the 3D scene, we propose an aligning algorithm which harmoniously integrates the portions of newly generated 3D scenes. The finally obtained 3D scene serves as initial points for optimizing Gaussian splats. LucidDreamer produces Gaussian splats that are highly-detailed compared to the previous 3D scene generation methods, with no constraint on domain of the target scene.