En3D：一個增強型生成模型，用於從2D合成數據雕塑3D人體

En3D: An Enhanced Generative Model for Sculpting 3D Humans from 2D Synthetic Data

January 2, 2024

作者: Yifang Men, Biwen Lei, Yuan Yao, Miaomiao Cui, Zhouhui Lian, Xuansong Xie

cs.AI

摘要

我們提出了En3D，一種增強型生成方案，用於雕塑高質量的3D人類頭像。與先前依賴稀缺3D數據集或具有不平衡觀看角度和不精確姿勢先驗的有限2D集合的作品不同，我們的方法旨在開發一種零樣本3D生成方案，能夠製造出外觀逼真、幾何精確且內容多樣的3D人類，而無需依賴預先存在的3D或2D資產。為應對這一挑戰，我們引入了一個精心設計的工作流程，從合成的2D數據中實現準確的物理建模，以學習增強的3D生成模型。在推斷期間，我們整合了優化模塊，以彌合逼真外觀與粗糙3D形狀之間的差距。具體而言，En3D包括三個模塊：一個3D生成器，從合成的平衡、多樣和結構化的人類圖像中準確建模可通用的3D人類，具有逼真外觀；一個幾何雕塑師，使用多視圖法向約束增強形狀質量，以處理複雜的人體解剖；以及一個紋理模塊，通過利用語義UV分割和可微分光柵化器，將明確的紋理映射解開，具有忠實度和可編輯性。實驗結果顯示，我們的方法在圖像質量、幾何精度和內容多樣性方面顯著優於先前的作品。我們還展示了我們生成的頭像在動畫和編輯中的應用性，以及我們的方法在內容風格自由適應方面的可擴展性。

English

We present En3D, an enhanced generative scheme for sculpting high-quality 3D human avatars. Unlike previous works that rely on scarce 3D datasets or limited 2D collections with imbalanced viewing angles and imprecise pose priors, our approach aims to develop a zero-shot 3D generative scheme capable of producing visually realistic, geometrically accurate and content-wise diverse 3D humans without relying on pre-existing 3D or 2D assets. To address this challenge, we introduce a meticulously crafted workflow that implements accurate physical modeling to learn the enhanced 3D generative model from synthetic 2D data. During inference, we integrate optimization modules to bridge the gap between realistic appearances and coarse 3D shapes. Specifically, En3D comprises three modules: a 3D generator that accurately models generalizable 3D humans with realistic appearance from synthesized balanced, diverse, and structured human images; a geometry sculptor that enhances shape quality using multi-view normal constraints for intricate human anatomy; and a texturing module that disentangles explicit texture maps with fidelity and editability, leveraging semantical UV partitioning and a differentiable rasterizer. Experimental results show that our approach significantly outperforms prior works in terms of image quality, geometry accuracy and content diversity. We also showcase the applicability of our generated avatars for animation and editing, as well as the scalability of our approach for content-style free adaptation.