Pandora3D：高品質3D形狀與紋理生成的綜合框架

Pandora3D: A Comprehensive Framework for High-Quality 3D Shape and Texture Generation

February 20, 2025

作者: Jiayu Yang, Taizhang Shang, Weixuan Sun, Xibin Song, Ziang Cheng, Senbo Wang, Shenzhou Chen, Weizhe Liu, Hongdong Li, Pan Ji

cs.AI

摘要

本報告提出了一個全面的框架，用於從多樣化的輸入提示（包括單張圖像、多視角圖像及文本描述）生成高品質的3D形狀與紋理。該框架涵蓋了3D形狀生成與紋理生成兩大部分。(1) 3D形狀生成流程採用變分自編碼器（VAE）將隱式3D幾何編碼至潛在空間，並利用擴散網絡根據輸入提示生成潛在變量，同時通過改進提升模型能力。此外，還探索了一種藝術家創建網格（AM）的生成方法，在處理較為簡單的幾何形狀時展現出良好前景。(2) 紋理生成則是一個多階段過程，始於正面圖像的生成，繼而進行多視角圖像生成、RGB到PBR紋理轉換，以及高分辨率多視角紋理精細化處理。每一階段均嵌入了一致性調度器，以在推理過程中強制多視角紋理間的像素級一致性，確保無縫整合。 該流程展現了對多種輸入格式的有效處理能力，利用先進的神經網絡架構與創新方法，產出高品質的3D內容。本報告詳細闡述了系統架構、實驗結果，以及未來改進與擴展框架的潛在方向。源代碼及預訓練權重已發佈於：https://github.com/Tencent/Tencent-XR-3DGen。

English

This report presents a comprehensive framework for generating high-quality 3D shapes and textures from diverse input prompts, including single images, multi-view images, and text descriptions. The framework consists of 3D shape generation and texture generation. (1). The 3D shape generation pipeline employs a Variational Autoencoder (VAE) to encode implicit 3D geometries into a latent space and a diffusion network to generate latents conditioned on input prompts, with modifications to enhance model capacity. An alternative Artist-Created Mesh (AM) generation approach is also explored, yielding promising results for simpler geometries. (2). Texture generation involves a multi-stage process starting with frontal images generation followed by multi-view images generation, RGB-to-PBR texture conversion, and high-resolution multi-view texture refinement. A consistency scheduler is plugged into every stage, to enforce pixel-wise consistency among multi-view textures during inference, ensuring seamless integration. The pipeline demonstrates effective handling of diverse input formats, leveraging advanced neural architectures and novel methodologies to produce high-quality 3D content. This report details the system architecture, experimental results, and potential future directions to improve and expand the framework. The source code and pretrained weights are released at: https://github.com/Tencent/Tencent-XR-3DGen.