DreamCraft3D：具有啟動擴散的分層3D生成

DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior

October 25, 2023

作者: Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, Yebin Liu

cs.AI

摘要

我們提出了DreamCraft3D，一種分層式3D內容生成方法，可產生高保真度和連貫的3D物體。我們通過利用2D參考圖像來引導幾何雕塑和紋理增強階段來解決這個問題。本研究的核心是解決現有作品遇到的一致性問題。為了雕塑呈現連貫的幾何形狀，我們通過基於視角的擴散模型執行分數蒸餾取樣。這種3D先驗連同幾種訓練策略，優先考慮幾何一致性，但會犧牲紋理保真度。我們進一步提出了引導分數蒸餾以專門增強紋理。我們在場景的擴增渲染上訓練了一個個性化的擴散模型Dreambooth，賦予其對正在優化的場景的3D知識。從這種3D感知擴散先驗中的分數蒸餾為場景提供了視角一致的引導。值得注意的是，通過交替優化擴散先驗和3D場景表示，我們實現了相互加強的改進：優化的3D場景有助於訓練特定場景的擴散模型，為3D優化提供越來越視角一致的引導。因此，優化是自我啟動的，並且導致了實質性的紋理增強。通過在分層生成過程中定制的3D先驗，DreamCraft3D生成具有照片般逼真渲染的連貫3D物體，推動了3D內容生成的最新技術。代碼可在https://github.com/deepseek-ai/DreamCraft3D找到。

English

We present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation. Code available at https://github.com/deepseek-ai/DreamCraft3D.