HyperDreamer：從單張圖像生成和編輯超逼真的3D內容

HyperDreamer: Hyper-Realistic 3D Content Generation and Editing from a Single Image

December 7, 2023

作者: Tong Wu, Zhibing Li, Shuai Yang, Pan Zhang, Xinggang Pan, Jiaqi Wang, Dahua Lin, Ziwei Liu

cs.AI

摘要

從單張圖片創建3D內容是一項歷史悠久且極具吸引力的任務。最近的進展引入了2D擴散先驗，產生了合理的結果。然而，現有方法對於後期生成的用途並不足以超現實，因為用戶無法從完整範圍查看、渲染和編輯生成的3D內容。為應對這些挑戰，我們引入了HyperDreamer，具有幾個關鍵設計和吸引人的特性：1)可查看：具有高分辨率紋理的360度網格建模使得可以從完整的觀察點創建視覺上引人入勝的3D模型。2)可渲染：細粒度的語義分割和數據驅動的先驗被納入指導，以學習合理的反照率、粗糙度和高光特性，實現語義感知任意材料估計。3)可編輯：對於生成的模型或用戶自己的數據，用戶可以通過幾次點擊互動選擇任何區域，並通過基於文本的指導高效編輯紋理。大量實驗證明了HyperDreamer在建模具有高分辨率紋理的區域感知材料和實現用戶友好編輯方面的有效性。我們相信HyperDreamer有望推動3D內容創建的發展並在各個領域找到應用。

English

3D content creation from a single image is a long-standing yet highly desirable task. Recent advances introduce 2D diffusion priors, yielding reasonable results. However, existing methods are not hyper-realistic enough for post-generation usage, as users cannot view, render and edit the resulting 3D content from a full range. To address these challenges, we introduce HyperDreamer with several key designs and appealing properties: 1) Viewable: 360 degree mesh modeling with high-resolution textures enables the creation of visually compelling 3D models from a full range of observation points. 2) Renderable: Fine-grained semantic segmentation and data-driven priors are incorporated as guidance to learn reasonable albedo, roughness, and specular properties of the materials, enabling semantic-aware arbitrary material estimation. 3) Editable: For a generated model or their own data, users can interactively select any region via a few clicks and efficiently edit the texture with text-based guidance. Extensive experiments demonstrate the effectiveness of HyperDreamer in modeling region-aware materials with high-resolution textures and enabling user-friendly editing. We believe that HyperDreamer holds promise for advancing 3D content creation and finding applications in various domains.