GaussianDreamerPro：具有高度增強品質的可操控3D高斯文本

GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Quality

June 26, 2024

作者: Taoran Yi, Jiemin Fang, Zanwei Zhou, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Xinggang Wang, Qi Tian

cs.AI

摘要

最近，3D 高斯濺射（3D-GS）在重建和渲染現實場景方面取得了巨大成功。為了將高渲染質量應用於生成任務，一系列研究作品試圖從文本生成 3D 高斯資產。然而，生成的資產並未達到重建任務中的質量水準。我們觀察到，由於生成過程可能導致不確定性，高斯往往會無節制地增長。為了極大地提升生成質量，我們提出了一個名為 GaussianDreamerPro 的新框架。其主要思想是將高斯綁定到合理的幾何形狀上，並在整個生成過程中進化。在我們框架的不同階段，幾何形狀和外觀都可以逐步豐富。最終輸出的資產是由綁定到網格的 3D 高斯構成，與先前方法相比展現出顯著增強的細節和質量。值得注意的是，生成的資產還可以無縫集成到下游操作流程中，例如動畫、合成和模擬等，極大地提升了其在各種應用中的潛力。演示可在 https://taoranyi.com/gaussiandreamerpro/ 查看。

English

Recently, 3D Gaussian splatting (3D-GS) has achieved great success in reconstructing and rendering real-world scenes. To transfer the high rendering quality to generation tasks, a series of research works attempt to generate 3D-Gaussian assets from text. However, the generated assets have not achieved the same quality as those in reconstruction tasks. We observe that Gaussians tend to grow without control as the generation process may cause indeterminacy. Aiming at highly enhancing the generation quality, we propose a novel framework named GaussianDreamerPro. The main idea is to bind Gaussians to reasonable geometry, which evolves over the whole generation process. Along different stages of our framework, both the geometry and appearance can be enriched progressively. The final output asset is constructed with 3D Gaussians bound to mesh, which shows significantly enhanced details and quality compared with previous methods. Notably, the generated asset can also be seamlessly integrated into downstream manipulation pipelines, e.g. animation, composition, and simulation etc., greatly promoting its potential in wide applications. Demos are available at https://taoranyi.com/gaussiandreamerpro/.