清晰梦境生成器：无域生成3D高斯飞溅场景

摘要

随着虚拟现实设备和内容的广泛使用，对3D场景生成技术的需求变得更加普遍。然而，现有的3D场景生成模型将目标场景限制在特定领域，主要是由于它们使用的3D扫描数据集训练策略远离真实世界。为了解决这种限制，我们提出了LucidDreamer，这是一个无域场景生成流程，充分利用现有大规模扩散生成模型的能力。我们的LucidDreamer有两个交替步骤：梦境和对齐。首先，为了从输入生成多视角一致的图像，我们将点云作为每个图像生成的几何指导线。具体来说，我们将点云的一部分投影到所需视角，并将投影作为生成模型进行修补时的指导。修补后的图像通过估计的深度图提升到3D空间，形成新的点。其次，为了将新点聚合到3D场景中，我们提出了一种对齐算法，可以和谐地整合新生成的3D场景部分。最终获得的3D场景作为优化高斯斑点的初始点。LucidDreamer生成的高斯斑点与先前的3D场景生成方法相比更加详细，且不受目标场景领域的约束。

English

With the widespread usage of VR devices and contents, demands for 3D scene generation techniques become more popular. Existing 3D scene generation models, however, limit the target scene to specific domain, primarily due to their training strategies using 3D scan dataset that is far from the real-world. To address such limitation, we propose LucidDreamer, a domain-free scene generation pipeline by fully leveraging the power of existing large-scale diffusion-based generative model. Our LucidDreamer has two alternate steps: Dreaming and Alignment. First, to generate multi-view consistent images from inputs, we set the point cloud as a geometrical guideline for each image generation. Specifically, we project a portion of point cloud to the desired view and provide the projection as a guidance for inpainting using the generative model. The inpainted images are lifted to 3D space with estimated depth maps, composing a new points. Second, to aggregate the new points into the 3D scene, we propose an aligning algorithm which harmoniously integrates the portions of newly generated 3D scenes. The finally obtained 3D scene serves as initial points for optimizing Gaussian splats. LucidDreamer produces Gaussian splats that are highly-detailed compared to the previous 3D scene generation methods, with no constraint on domain of the target scene.

清晰梦境生成器：无域生成3D高斯飞溅场景

LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes

摘要

Support