DreamWaltz-G：从骨架引导的2D扩散生成具有表现力的3D高斯化身

DreamWaltz-G: Expressive 3D Gaussian Avatars from Skeleton-Guided 2D Diffusion

September 25, 2024

作者: Yukun Huang, Jianan Wang, Ailing Zeng, Zheng-Jun Zha, Lei Zhang, Xihui Liu

cs.AI

摘要

利用预训练的2D扩散模型和分数蒸馏采样（SDS），最近的方法展示了文本到3D头像生成方面的有希望的结果。然而，生成具有表现力动画能力的高质量3D头像仍然具有挑战性。在这项工作中，我们提出了DreamWaltz-G，一个用于从文本生成可动画的3D头像的新型学习框架。该框架的核心在于基于骨骼引导的分数蒸馏和混合3D高斯头像表示。具体而言，所提出的骨骼引导分数蒸馏将3D人体模板的骨骼控制集成到2D扩散模型中，增强了在视角和人体姿势方面的SDS监督的一致性。这有助于生成高质量的头像，减轻了诸如多个面部、额外肢体和模糊等问题。所提出的混合3D高斯头像表示建立在高效的3D高斯基础上，结合了神经隐式场和参数化的3D网格，实现了实时渲染、稳定的SDS优化和表现力动画。大量实验证明DreamWaltz-G在生成和动画化3D头像方面非常有效，在视觉质量和动画表现力方面优于现有方法。我们的框架进一步支持各种应用，包括人类视频再现和多主体场景合成。

English

Leveraging pretrained 2D diffusion models and score distillation sampling (SDS), recent methods have shown promising results for text-to-3D avatar generation. However, generating high-quality 3D avatars capable of expressive animation remains challenging. In this work, we present DreamWaltz-G, a novel learning framework for animatable 3D avatar generation from text. The core of this framework lies in Skeleton-guided Score Distillation and Hybrid 3D Gaussian Avatar representation. Specifically, the proposed skeleton-guided score distillation integrates skeleton controls from 3D human templates into 2D diffusion models, enhancing the consistency of SDS supervision in terms of view and human pose. This facilitates the generation of high-quality avatars, mitigating issues such as multiple faces, extra limbs, and blurring. The proposed hybrid 3D Gaussian avatar representation builds on the efficient 3D Gaussians, combining neural implicit fields and parameterized 3D meshes to enable real-time rendering, stable SDS optimization, and expressive animation. Extensive experiments demonstrate that DreamWaltz-G is highly effective in generating and animating 3D avatars, outperforming existing methods in both visual quality and animation expressiveness. Our framework further supports diverse applications, including human video reenactment and multi-subject scene composition.