TripoSG：使用大規模矯正流模型進行高保真度的3D形狀合成

TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Models

February 10, 2025

作者: Yangguang Li, Zi-Xin Zou, Zexiang Liu, Dehu Wang, Yuan Liang, Zhipeng Yu, Xingchao Liu, Yuan-Chen Guo, Ding Liang, Wanli Ouyang, Yan-Pei Cao

cs.AI

摘要

最近擴散技術的進步推動了影像和視頻生成達到前所未有的質量水平，顯著加快了生成式人工智慧的部署和應用。然而，3D形狀生成技術迄今仍然落後，受到3D數據規模的限制、3D數據處理的複雜性以及對3D領域先進技術的不足探索所限制。目前的3D形狀生成方法在輸出質量、泛化能力和與輸入條件的對齊方面面臨著重大挑戰。我們提出了TripoSG，一種新的簡化形狀擴散範式，能夠生成與輸入圖像精確對應的高保真度3D網格。具體來說，我們提出：1）一種用於3D形狀生成的大規模矯正流轉換器，通過在大量高質量數據上進行訓練實現了最先進的保真度。2）一種結合SDF、法線和eikonal損失的混合監督訓練策略，用於3D VAE，實現了高質量的3D重建性能。3）一個數據處理流水線，用於生成200萬個高質量3D樣本，突出了在訓練3D生成模型時數據質量和數量的關鍵規則。通過全面的實驗，我們驗證了我們新框架中每個組件的有效性。這些部分的無縫集成使TripoSG在3D形狀生成方面實現了最先進的性能。由於高分辨率能力，生成的3D形狀展示了增強的細節，並且對輸入圖像表現出卓越的保真度。此外，TripoSG展示了在從不同圖像風格和內容生成3D模型方面的改進多樣性，展示了強大的泛化能力。為了促進3D生成領域的進步和創新，我們將使我們的模型公開可用。

English

Recent advancements in diffusion techniques have propelled image and video generation to unprece- dented levels of quality, significantly accelerating the deployment and application of generative AI. However, 3D shape generation technology has so far lagged behind, constrained by limitations in 3D data scale, complexity of 3D data process- ing, and insufficient exploration of advanced tech- niques in the 3D domain. Current approaches to 3D shape generation face substantial challenges in terms of output quality, generalization capa- bility, and alignment with input conditions. We present TripoSG, a new streamlined shape diffu- sion paradigm capable of generating high-fidelity 3D meshes with precise correspondence to input images. Specifically, we propose: 1) A large-scale rectified flow transformer for 3D shape generation, achieving state-of-the-art fidelity through training on extensive, high-quality data. 2) A hybrid supervised training strategy combining SDF, normal, and eikonal losses for 3D VAE, achieving high- quality 3D reconstruction performance. 3) A data processing pipeline to generate 2 million high- quality 3D samples, highlighting the crucial rules for data quality and quantity in training 3D gen- erative models. Through comprehensive experi- ments, we have validated the effectiveness of each component in our new framework. The seamless integration of these parts has enabled TripoSG to achieve state-of-the-art performance in 3D shape generation. The resulting 3D shapes exhibit en- hanced detail due to high-resolution capabilities and demonstrate exceptional fidelity to input im- ages. Moreover, TripoSG demonstrates improved versatility in generating 3D models from diverse image styles and contents, showcasing strong gen- eralization capabilities. To foster progress and innovation in the field of 3D generation, we will make our model publicly available.