在文本生成3D过程中，控制得分蒸馏中的模式坍缩

Taming Mode Collapse in Score Distillation for Text-to-3D Generation

December 31, 2023

作者: Peihao Wang, Dejia Xu, Zhiwen Fan, Dilin Wang, Sreyas Mohan, Forrest Iandola, Rakesh Ranjan, Yilei Li, Qiang Liu, Zhangyang Wang, Vikas Chandra

cs.AI

摘要

尽管得分蒸馏在文本到3D生成中表现出色，但这些技术因存在视角不一致问题而臭名昭著，也被称为“雅努斯”伪影现象，即生成的物体在多个视角上都具有多个正面。尽管经验有效的方法已经通过得分去偏倚或提示工程来解决这一问题，但对于解释和解决这一问题的更严格视角仍然难以捉摸。在本文中，我们揭示现有基于得分蒸馏的文本到3D生成框架在每个视角上都退化为最大似然寻求，因此在实践中出现了模式坍塌问题，表现为雅努斯伪影现象。为了遏制模式坍塌，我们通过在相应变分目标中重新引入熵项来改进得分蒸馏，该熵项应用于渲染图像的分布。最大化熵鼓励在生成的3D资产的不同视角之间保持多样性，从而缓解雅努斯问题。基于这一新目标，我们推导出一种新的3D得分蒸馏更新规则，称为熵得分蒸馏（ESD）。我们在理论上揭示，ESD可以通过仅采用基于分类器的自由引导技巧来简化和实现变分得分蒸馏。尽管这一方法非常直接，但我们的大量实验成功地证明ESD可以有效地处理得分蒸馏中的雅努斯伪影现象。

English

Despite the remarkable performance of score distillation in text-to-3D generation, such techniques notoriously suffer from view inconsistency issues, also known as "Janus" artifact, where the generated objects fake each view with multiple front faces. Although empirically effective methods have approached this problem via score debiasing or prompt engineering, a more rigorous perspective to explain and tackle this problem remains elusive. In this paper, we reveal that the existing score distillation-based text-to-3D generation frameworks degenerate to maximal likelihood seeking on each view independently and thus suffer from the mode collapse problem, manifesting as the Janus artifact in practice. To tame mode collapse, we improve score distillation by re-establishing in entropy term in the corresponding variational objective, which is applied to the distribution of rendered images. Maximizing the entropy encourages diversity among different views in generated 3D assets, thereby mitigating the Janus problem. Based on this new objective, we derive a new update rule for 3D score distillation, dubbed Entropic Score Distillation (ESD). We theoretically reveal that ESD can be simplified and implemented by just adopting the classifier-free guidance trick upon variational score distillation. Although embarrassingly straightforward, our extensive experiments successfully demonstrate that ESD can be an effective treatment for Janus artifacts in score distillation.