Hi3D：利用視頻擴散模型追求高分辨率圖像到三維生成

Hi3D: Pursuing High-Resolution Image-to-3D Generation with Video Diffusion Models

September 11, 2024

作者: Haibo Yang, Yang Chen, Yingwei Pan, Ting Yao, Zhineng Chen, Chong-Wah Ngo, Tao Mei

cs.AI

摘要

儘管在圖像轉3D生成方面取得了巨大進展，現有方法仍然難以產生具有高分辨率細節的多視角一致圖像紋理，尤其是在缺乏3D意識的2D擴散範式中。在這項工作中，我們提出了高分辨率圖像轉3D模型（Hi3D），這是一種基於新的視頻擴散範式，重新定義了將單張圖像轉換為多視角圖像，作為具有3D意識的連續圖像生成（即軌道視頻生成）。該方法深入探討了視頻擴散模型中的基礎時間一致性知識，這對於在3D生成中跨多個視角實現幾何一致性具有良好的泛化能力。從技術上講，Hi3D首先通過3D意識先驗（相機姿態條件）賦予預訓練的視頻擴散模型能力，生成具有低分辨率紋理細節的多視角圖像。學習了一個3D意識的視頻對視頻精化器，進一步提高了具有高分辨率紋理細節的多視角圖像。這些高分辨率多視角圖像通過3D高斯飛濺進行新視角擴增，最終通過3D重建獲得高保真度網格。對於新視角合成和單視角重建的大量實驗表明，我們的Hi3D成功生成了具有高度細節紋理的優質多視角一致性圖像。源代碼和數據可在https://github.com/yanghb22-fdu/Hi3D-Official找到。

English

Despite having tremendous progress in image-to-3D generation, existing methods still struggle to produce multi-view consistent images with high-resolution textures in detail, especially in the paradigm of 2D diffusion that lacks 3D awareness. In this work, we present High-resolution Image-to-3D model (Hi3D), a new video diffusion based paradigm that redefines a single image to multi-view images as 3D-aware sequential image generation (i.e., orbital video generation). This methodology delves into the underlying temporal consistency knowledge in video diffusion model that generalizes well to geometry consistency across multiple views in 3D generation. Technically, Hi3D first empowers the pre-trained video diffusion model with 3D-aware prior (camera pose condition), yielding multi-view images with low-resolution texture details. A 3D-aware video-to-video refiner is learnt to further scale up the multi-view images with high-resolution texture details. Such high-resolution multi-view images are further augmented with novel views through 3D Gaussian Splatting, which are finally leveraged to obtain high-fidelity meshes via 3D reconstruction. Extensive experiments on both novel view synthesis and single view reconstruction demonstrate that our Hi3D manages to produce superior multi-view consistency images with highly-detailed textures. Source code and data are available at https://github.com/yanghb22-fdu/Hi3D-Official.