DiffusionGAN3D：通过结合3D GAN和扩散先验来增强文本引导的3D生成和领域自适应。

DiffusionGAN3D: Boosting Text-guided 3D Generation and Domain Adaption by Combining 3D GANs and Diffusion Priors

December 28, 2023

作者: Biwen Lei, Kai Yu, Mengyang Feng, Miaomiao Cui, Xuansong Xie

cs.AI

摘要

文本引导的领域自适应和生成具有3D感知的肖像在各个领域中有许多应用。然而，由于缺乏训练数据以及处理高度多样的几何和外观方面的挑战，针对这些任务的现有方法存在着诸如缺乏灵活性、不稳定性和低保真度等问题。在本文中，我们提出了一个新颖的框架DiffusionGAN3D，通过结合3D GANs和扩散先验来增强文本引导的3D领域自适应和生成。具体而言，我们集成了预训练的3D生成模型（例如EG3D）和文本到图像扩散模型。前者为从文本生成稳定且高质量的头像提供了坚实基础。而扩散模型则提供强大的先验，并指导3D生成器以信息丰富的方向进行微调，以实现灵活且高效的文本引导领域自适应。为了增强领域自适应中的多样性和文本到头像生成能力，我们分别引入了相对距离损失和特定案例可学习的三平面。此外，我们设计了一个渐进式纹理细化模块，以提高上述两个任务的纹理质量。大量实验证明，所提出的框架在领域自适应和文本到头像任务中取得了出色的结果，在生成质量和效率方面优于现有方法。项目主页位于https://younglbw.github.io/DiffusionGAN3D-homepage/.

English

Text-guided domain adaption and generation of 3D-aware portraits find many applications in various fields. However, due to the lack of training data and the challenges in handling the high variety of geometry and appearance, the existing methods for these tasks suffer from issues like inflexibility, instability, and low fidelity. In this paper, we propose a novel framework DiffusionGAN3D, which boosts text-guided 3D domain adaption and generation by combining 3D GANs and diffusion priors. Specifically, we integrate the pre-trained 3D generative models (e.g., EG3D) and text-to-image diffusion models. The former provides a strong foundation for stable and high-quality avatar generation from text. And the diffusion models in turn offer powerful priors and guide the 3D generator finetuning with informative direction to achieve flexible and efficient text-guided domain adaption. To enhance the diversity in domain adaption and the generation capability in text-to-avatar, we introduce the relative distance loss and case-specific learnable triplane respectively. Besides, we design a progressive texture refinement module to improve the texture quality for both tasks above. Extensive experiments demonstrate that the proposed framework achieves excellent results in both domain adaption and text-to-avatar tasks, outperforming existing methods in terms of generation quality and efficiency. The project homepage is at https://younglbw.github.io/DiffusionGAN3D-homepage/.