LayerPano3D：用于超沉浸式场景生成的分层3D全景

LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation

August 23, 2024

作者: Shuai Yang, Jing Tan, Mengchen Zhang, Tong Wu, Yixuan Li, Gordon Wetzstein, Ziwei Liu, Dahua Lin

cs.AI

摘要

在计算机视觉和图形学中，3D沉浸式场景生成是一项具有挑战性但又至关重要的任务。理想的虚拟3D场景应该具备以下特点：1）展现全方位视角的一致性，2）允许在复杂场景层次结构中自由探索。现有方法要么依赖于通过修补逐步扩展场景，要么采用全景表示来表达大视场场景环境。然而，在扩展过程中生成的场景容易出现语义漂移，并且无法处理场景层次结构中的遮挡。为了解决这些挑战，我们引入了LayerPano3D，这是一个新颖的框架，用于从单个文本提示生成全景式的可探索的3D场景。我们的关键洞察是将参考的2D全景图分解成不同深度级别的多个层，其中每个层通过扩散先验从参考视角中展示未见空间。LayerPano3D包括多个专门设计：1）我们引入了一种新颖的文本引导的锚定视图合成流程，用于高质量、一致的全景生成。2）我们开创了作为底层表示的分层3D全景，用于管理复杂的场景层次结构，并将其提升为3D高斯函数，以涂抹详细的360度全方位场景，具有不受限制的观看路径。大量实验证明，我们的框架在全景一致性和沉浸式探索体验方面生成了最先进的3D全景场景。我们相信LayerPano3D在推进3D全景场景创建方面具有广泛的应用前景。

English

3D immersive scene generation is a challenging yet critical task in computer vision and graphics. A desired virtual 3D scene should 1) exhibit omnidirectional view consistency, and 2) allow for free exploration in complex scene hierarchies. Existing methods either rely on successive scene expansion via inpainting or employ panorama representation to represent large FOV scene environments. However, the generated scene suffers from semantic drift during expansion and is unable to handle occlusion among scene hierarchies. To tackle these challenges, we introduce LayerPano3D, a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt. Our key insight is to decompose a reference 2D panorama into multiple layers at different depth levels, where each layer reveals the unseen space from the reference views via diffusion prior. LayerPano3D comprises multiple dedicated designs: 1) we introduce a novel text-guided anchor view synthesis pipeline for high-quality, consistent panorama generation. 2) We pioneer the Layered 3D Panorama as underlying representation to manage complex scene hierarchies and lift it into 3D Gaussians to splat detailed 360-degree omnidirectional scenes with unconstrained viewing paths. Extensive experiments demonstrate that our framework generates state-of-the-art 3D panoramic scene in both full view consistency and immersive exploratory experience. We believe that LayerPano3D holds promise for advancing 3D panoramic scene creation with numerous applications.