ProlificDreamer：通过变分分数蒸馏实现高保真度和多样性的文本到3D生成

ProlificDreamer: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation

May 25, 2023

作者: Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, Jun Zhu

cs.AI

摘要

得分蒸馏采样（SDS）在文本到三维生成中表现出巨大潜力，通过蒸馏预训练的大规模文本到图像扩散模型，但存在过饱和、过平滑和低多样性问题。在这项工作中，我们建议将三维参数建模为随机变量，而不是像在SDS中那样作为常数，并提出变分得分蒸馏（VSD），这是一个基于粒子的变分框架，用于解释和解决文本到三维生成中提到的问题。我们展示了SDS是VSD的一个特例，并导致具有小和大CFG权重的低质量样本。相比之下，VSD在各种CFG权重下表现良好，作为从扩散模型中的祖先采样，同时通过常见的CFG权重（即7.5）提高了多样性和样本质量。我们进一步提出了文本到三维设计空间的各种改进，如蒸馏时间表和密度初始化，这些改进与蒸馏算法正交，但尚未得到很好的探索。我们的整体方法，命名为ProlificDreamer，可以生成高渲染分辨率（即512x512）和高保真度的NeRF，具有丰富的结构和复杂效果（如烟雾和水滴）。此外，由NeRF初始化，通过VSD微调的网格精细详细且逼真。项目页面：https://ml.cs.tsinghua.edu.cn/prolificdreamer/

English

Score distillation sampling (SDS) has shown great promise in text-to-3D generation by distilling pretrained large-scale text-to-image diffusion models, but suffers from over-saturation, over-smoothing, and low-diversity problems. In this work, we propose to model the 3D parameter as a random variable instead of a constant as in SDS and present variational score distillation (VSD), a principled particle-based variational framework to explain and address the aforementioned issues in text-to-3D generation. We show that SDS is a special case of VSD and leads to poor samples with both small and large CFG weights. In comparison, VSD works well with various CFG weights as ancestral sampling from diffusion models and simultaneously improves the diversity and sample quality with a common CFG weight (i.e., 7.5). We further present various improvements in the design space for text-to-3D such as distillation time schedule and density initialization, which are orthogonal to the distillation algorithm yet not well explored. Our overall approach, dubbed ProlificDreamer, can generate high rendering resolution (i.e., 512times512) and high-fidelity NeRF with rich structure and complex effects (e.g., smoke and drops). Further, initialized from NeRF, meshes fine-tuned by VSD are meticulously detailed and photo-realistic. Project page: https://ml.cs.tsinghua.edu.cn/prolificdreamer/