AGG：攤銷生成式 3D 高斯模型，用於從單張圖像到 3D 的轉換

AGG: Amortized Generative 3D Gaussians for Single Image to 3D

January 8, 2024

作者: Dejia Xu, Ye Yuan, Morteza Mardani, Sifei Liu, Jiaming Song, Zhangyang Wang, Arash Vahdat

cs.AI

摘要

隨著對自動3D內容創建流程需求的增加，已經研究了各種3D表示形式，以從單張圖像生成3D物體。由於其優越的渲染效率，基於3D高斯擴散的模型最近在3D重建和生成方面表現出色。基於3D高斯擴散的方法用於從圖像生成3D的過程通常是基於優化的，需要進行許多計算昂貴的分數提煉步驟。為了克服這些挑戰，我們引入了一種攤銷生成式3D高斯框架（AGG），可以即時從單張圖像生成3D高斯，無需進行每個實例的優化。通過使用中間混合表示，AGG將3D高斯位置的生成和其他外觀屬性的聯合優化進行了分解。此外，我們提出了一個分級流程，首先生成3D數據的粗略表示，然後再通過3D高斯超分辨率模塊對其進行上採樣。我們的方法與現有基於優化的3D高斯框架和使用其他3D表示的基於採樣的流程進行了評估，其中AGG在質量和量化方面展示出競爭力的生成能力，同時速度快了幾個數量級。項目頁面：https://ir1d.github.io/AGG/

English

Given the growing need for automatic 3D content creation pipelines, various 3D representations have been studied to generate 3D objects from a single image. Due to its superior rendering efficiency, 3D Gaussian splatting-based models have recently excelled in both 3D reconstruction and generation. 3D Gaussian splatting approaches for image to 3D generation are often optimization-based, requiring many computationally expensive score-distillation steps. To overcome these challenges, we introduce an Amortized Generative 3D Gaussian framework (AGG) that instantly produces 3D Gaussians from a single image, eliminating the need for per-instance optimization. Utilizing an intermediate hybrid representation, AGG decomposes the generation of 3D Gaussian locations and other appearance attributes for joint optimization. Moreover, we propose a cascaded pipeline that first generates a coarse representation of the 3D data and later upsamples it with a 3D Gaussian super-resolution module. Our method is evaluated against existing optimization-based 3D Gaussian frameworks and sampling-based pipelines utilizing other 3D representations, where AGG showcases competitive generation abilities both qualitatively and quantitatively while being several orders of magnitude faster. Project page: https://ir1d.github.io/AGG/