DreamCraft3D：具有引导扩散的分层3D生成

DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion Prior

October 25, 2023

作者: Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, Yebin Liu

cs.AI

摘要

我们提出了DreamCraft3D，这是一种分层3D内容生成方法，能够产生高保真度和连贯性的3D物体。我们通过利用2D参考图像来引导几何塑造和纹理增强阶段，来解决这一问题。本研究的核心是解决现有作品遇到的一致性问题。为了塑造呈现连贯性的几何形状，我们通过视角相关扩散模型执行得分蒸馏采样。这种3D先验模型，连同几种训练策略，优先考虑几何一致性，但会牺牲纹理保真度。我们进一步提出了引导得分蒸馏，专门用于增强纹理。我们在场景的增强渲染上训练了一个个性化扩散模型Dreambooth，使其具备对正在优化的场景的3D知识。从这种3D感知扩散先验中蒸馏得分为场景提供了视角一致的指导。值得注意的是，通过交替优化扩散先验和3D场景表示，我们实现了相互增强的改进：优化的3D场景有助于训练特定场景的扩散模型，为3D优化提供日益视角一致的指导。因此，优化是自启动的，并且导致了大幅度的纹理增强。通过在分层生成过程中定制的3D先验，DreamCraft3D生成了具有逼真渲染的连贯3D物体，推动了3D内容生成技术的最新发展。代码可在https://github.com/deepseek-ai/DreamCraft3D找到。

English

We present DreamCraft3D, a hierarchical 3D content generation method that produces high-fidelity and coherent 3D objects. We tackle the problem by leveraging a 2D reference image to guide the stages of geometry sculpting and texture boosting. A central focus of this work is to address the consistency issue that existing works encounter. To sculpt geometries that render coherently, we perform score distillation sampling via a view-dependent diffusion model. This 3D prior, alongside several training strategies, prioritizes the geometry consistency but compromises the texture fidelity. We further propose Bootstrapped Score Distillation to specifically boost the texture. We train a personalized diffusion model, Dreambooth, on the augmented renderings of the scene, imbuing it with 3D knowledge of the scene being optimized. The score distillation from this 3D-aware diffusion prior provides view-consistent guidance for the scene. Notably, through an alternating optimization of the diffusion prior and 3D scene representation, we achieve mutually reinforcing improvements: the optimized 3D scene aids in training the scene-specific diffusion model, which offers increasingly view-consistent guidance for 3D optimization. The optimization is thus bootstrapped and leads to substantial texture boosting. With tailored 3D priors throughout the hierarchical generation, DreamCraft3D generates coherent 3D objects with photorealistic renderings, advancing the state-of-the-art in 3D content generation. Code available at https://github.com/deepseek-ai/DreamCraft3D.