4K4DGen：全景4K分辨率的4D生成。

4K4DGen: Panoramic 4D Generation at 4K Resolution

June 19, 2024

作者: Renjie Li, Panwang Pan, Bangbang Yang, Dejia Xu, Shijie Zhou, Xuanyang Zhang, Zeming Li, Achuta Kadambi, Zhangyang Wang, Zhiwen Fan

cs.AI

摘要

虚拟现实和增强现实（VR/AR）技术的蓬勃发展推动了对高质量、沉浸式和动态环境创造的需求不断增加。然而，现有的生成技术要么仅专注于动态对象，要么从单一视角图像进行外延，未能满足VR/AR应用的需求。在这项工作中，我们致力于将单个全景提升为沉浸式的4D体验这一具有挑战性的任务。我们首次展示了生成具有360度视角的全方位动态场景的能力，分辨率为4K，从而提供沉浸式用户体验。我们的方法引入了一个流程，促进自然场景动画，并利用高效的点阵技术优化一组4D高斯模型，以实现实时探索。为了克服在全景格式中缺乏场景尺度标注的4D数据和模型，我们提出了一种新颖的全景去噪器，将通用的2D扩散先验调整为在360度图像中一致地生成动画，将其转化为具有目标区域动态场景的全景视频。随后，我们将全景视频提升为一个保持空间和时间一致性的4D沉浸式环境。通过将来自透视域的2D模型的先验知识转移到全景域和具有空间外观和几何正则化的4D提升，我们首次实现了（4096乘以2048）分辨率下高质量的全景到4D生成。请访问项目网站https://4k4dgen.github.io。

English

The blooming of virtual reality and augmented reality (VR/AR) technologies has driven an increasing demand for the creation of high-quality, immersive, and dynamic environments. However, existing generative techniques either focus solely on dynamic objects or perform outpainting from a single perspective image, failing to meet the needs of VR/AR applications. In this work, we tackle the challenging task of elevating a single panorama to an immersive 4D experience. For the first time, we demonstrate the capability to generate omnidirectional dynamic scenes with 360-degree views at 4K resolution, thereby providing an immersive user experience. Our method introduces a pipeline that facilitates natural scene animations and optimizes a set of 4D Gaussians using efficient splatting techniques for real-time exploration. To overcome the lack of scene-scale annotated 4D data and models, especially in panoramic formats, we propose a novel Panoramic Denoiser that adapts generic 2D diffusion priors to animate consistently in 360-degree images, transforming them into panoramic videos with dynamic scenes at targeted regions. Subsequently, we elevate the panoramic video into a 4D immersive environment while preserving spatial and temporal consistency. By transferring prior knowledge from 2D models in the perspective domain to the panoramic domain and the 4D lifting with spatial appearance and geometry regularization, we achieve high-quality Panorama-to-4D generation at a resolution of (4096 times 2048) for the first time. See the project website at https://4k4dgen.github.io.