DreamCube：通过多平面同步实现3D全景图生成

摘要

三维全景合成是一项前景广阔但极具挑战性的任务，它要求生成的环视内容具备高质量且多样化的视觉外观与几何结构。现有方法通过利用预训练的二维基础模型中的丰富图像先验，来缓解三维全景数据稀缺的问题，然而三维全景与二维单视图之间的不兼容性限制了这些方法的效能。在本研究中，我们证明了通过对二维基础模型中的操作符应用多平面同步技术，能够将其能力无缝扩展至全向领域。基于这一设计，我们进一步提出了DreamCube，一个用于三维全景生成的多平面RGB-D扩散模型，该模型最大限度地复用二维基础模型的先验知识，以实现多样化的外观与精确的几何结构，同时保持多视角一致性。大量实验验证了我们的方法在全景图像生成、全景深度估计及三维场景生成中的有效性。

English

3D panorama synthesis is a promising yet challenging task that demands high-quality and diverse visual appearance and geometry of the generated omnidirectional content. Existing methods leverage rich image priors from pre-trained 2D foundation models to circumvent the scarcity of 3D panoramic data, but the incompatibility between 3D panoramas and 2D single views limits their effectiveness. In this work, we demonstrate that by applying multi-plane synchronization to the operators from 2D foundation models, their capabilities can be seamlessly extended to the omnidirectional domain. Based on this design, we further introduce DreamCube, a multi-plane RGB-D diffusion model for 3D panorama generation, which maximizes the reuse of 2D foundation model priors to achieve diverse appearances and accurate geometry while maintaining multi-view consistency. Extensive experiments demonstrate the effectiveness of our approach in panoramic image generation, panoramic depth estimation, and 3D scene generation.

DreamCube：通过多平面同步实现3D全景图生成

DreamCube: 3D Panorama Generation via Multi-plane Synchronization

摘要

Support