En3D：一种增强的生成模型，用于从2D合成数据雕塑3D人体

En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data

January 2, 2024

作者: Yifang Men, Biwen Lei, Yuan Yao, Miaomiao Cui, Zhouhui Lian, Xuansong Xie

cs.AI

摘要

我们提出了En3D，这是一种增强的生成方案，用于塑造高质量的3D人类化身。与先前依赖稀缺3D数据集或具有不平衡视角和不精确姿势先验的有限2D集合的作品不同，我们的方法旨在开发一种零样本3D生成方案，能够生成外观逼真、几何精确且内容多样的3D人类，而无需依赖现有的3D或2D资产。为了解决这一挑战，我们引入了一个精心设计的工作流程，通过实现精确的物理建模来从合成的2D数据中学习增强的3D生成模型。在推断过程中，我们集成了优化模块，以弥合逼真外观和粗糙3D形状之间的差距。具体而言，En3D包括三个模块：一个3D生成器，能够准确地建模出具有逼真外观的通用3D人类，从合成的平衡、多样化和结构化人类图像中获得；一个几何雕刻器，利用多视角法线约束增强形状质量，适用于复杂的人体解剖结构；以及一个纹理模块，通过语义UV分区和可微分光栅化器，将显式纹理映射解耦为具有保真度和可编辑性的纹理，实现了纹理的分离。实验结果表明，我们的方法在图像质量、几何精度和内容多样性方面明显优于先前的作品。我们还展示了我们生成的化身在动画和编辑方面的适用性，以及我们的方法在内容风格自由适应方面的可扩展性。

English

We present En3D, an enhanced generative scheme for sculpting high-quality 3D human avatars. Unlike previous works that rely on scarce 3D datasets or limited 2D collections with imbalanced viewing angles and imprecise pose priors, our approach aims to develop a zero-shot 3D generative scheme capable of producing visually realistic, geometrically accurate and content-wise diverse 3D humans without relying on pre-existing 3D or 2D assets. To address this challenge, we introduce a meticulously crafted workflow that implements accurate physical modeling to learn the enhanced 3D generative model from synthetic 2D data. During inference, we integrate optimization modules to bridge the gap between realistic appearances and coarse 3D shapes. Specifically, En3D comprises three modules: a 3D generator that accurately models generalizable 3D humans with realistic appearance from synthesized balanced, diverse, and structured human images; a geometry sculptor that enhances shape quality using multi-view normal constraints for intricate human anatomy; and a texturing module that disentangles explicit texture maps with fidelity and editability, leveraging semantical UV partitioning and a differentiable rasterizer. Experimental results show that our approach significantly outperforms prior works in terms of image quality, geometry accuracy and content diversity. We also showcase the applicability of our generated avatars for animation and editing, as well as the scalability of our approach for content-style free adaptation.