Hi3D：利用视频扩散模型实现高分辨率图像到三维生成

Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models

September 11, 2024

作者: Haibo Yang, Yang Chen, Yingwei Pan, Ting Yao, Zhineng Chen, Chong-Wah Ngo, Tao Mei

cs.AI

摘要

尽管在图像到3D生成方面取得了巨大进展，现有方法仍然难以生成具有高分辨率纹理细节的多视角一致图像，特别是在缺乏3D意识的2D扩散范式中。在这项工作中，我们提出了高分辨率图像到3D模型（Hi3D），这是一种基于视频扩散的新范式，将单个图像重新定义为多视角图像，作为具有3D意识的顺序图像生成（即轨道视频生成）。该方法深入研究了视频扩散模型中的基础时间一致性知识，这种知识在3D生成中能够很好地推广到多视角的几何一致性。从技术上讲，Hi3D首先通过3D意识先验（摄像机姿态条件）增强预训练的视频扩散模型，生成具有低分辨率纹理细节的多视角图像。然后学习了一种具有3D意识的视频到视频细化器，进一步扩大多视角图像的高分辨率纹理细节。这些高分辨率多视角图像通过3D高斯喷洒增加新颖视角，最终通过3D重建获得高保真度的网格。对新颖视角合成和单视角重建的大量实验表明，我们的Hi3D能够生成具有高度详细纹理的优质多视角一致图像。源代码和数据可在https://github.com/yanghb22-fdu/Hi3D-Official获取。

English

Despite having tremendous progress in image-to-3D generation, existing methods still struggle to produce multi-view consistent images with high-resolution textures in detail, especially in the paradigm of 2D diffusion that lacks 3D awareness. In this work, we present High-resolution Image-to-3D model (Hi3D), a new video diffusion based paradigm that redefines a single image to multi-view images as 3D-aware sequential image generation (i.e., orbital video generation). This methodology delves into the underlying temporal consistency knowledge in video diffusion model that generalizes well to geometry consistency across multiple views in 3D generation. Technically, Hi3D first empowers the pre-trained video diffusion model with 3D-aware prior (camera pose condition), yielding multi-view images with low-resolution texture details. A 3D-aware video-to-video refiner is learnt to further scale up the multi-view images with high-resolution texture details. Such high-resolution multi-view images are further augmented with novel views through 3D Gaussian Splatting, which are finally leveraged to obtain high-fidelity meshes via 3D reconstruction. Extensive experiments on both novel view synthesis and single view reconstruction demonstrate that our Hi3D manages to produce superior multi-view consistency images with highly-detailed textures. Source code and data are available at https://github.com/yanghb22-fdu/Hi3D-Official.