GaussianDreamerPro：具有高度增强质量的可操纵3D高斯文本

GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Quality

June 26, 2024

作者: Taoran Yi, Jiemin Fang, Zanwei Zhou, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Xinggang Wang, Qi Tian

cs.AI

摘要

最近，3D 高斯光滑（3D-GS）在重建和渲染现实场景方面取得了巨大成功。为了将高质量渲染转移到生成任务中，一系列研究工作尝试从文本生成 3D 高斯资产。然而，生成的资产未能达到重建任务中的质量水平。我们观察到，由于生成过程可能导致不确定性，高斯函数往往会无法控制地增长。为了极大地提升生成质量，我们提出了一种名为 GaussianDreamerPro 的新框架。其主要思想是将高斯函数绑定到合理的几何形状上，并在整个生成过程中演变。在我们框架的不同阶段，几何形状和外观都可以逐步丰富。最终输出的资产是由绑定到网格的 3D 高斯函数构建的，与先前方法相比，显示出显著增强的细节和质量。值得注意的是，生成的资产还可以无缝集成到下游处理流程中，例如动画、合成和模拟等，极大地提升了其在广泛应用中的潜力。演示可在 https://taoranyi.com/gaussiandreamerpro/ 上找到。

English

Recently, 3D Gaussian splatting (3D-GS) has achieved great success in reconstructing and rendering real-world scenes. To transfer the high rendering quality to generation tasks, a series of research works attempt to generate 3D-Gaussian assets from text. However, the generated assets have not achieved the same quality as those in reconstruction tasks. We observe that Gaussians tend to grow without control as the generation process may cause indeterminacy. Aiming at highly enhancing the generation quality, we propose a novel framework named GaussianDreamerPro. The main idea is to bind Gaussians to reasonable geometry, which evolves over the whole generation process. Along different stages of our framework, both the geometry and appearance can be enriched progressively. The final output asset is constructed with 3D Gaussians bound to mesh, which shows significantly enhanced details and quality compared with previous methods. Notably, the generated asset can also be seamlessly integrated into downstream manipulation pipelines, e.g. animation, composition, and simulation etc., greatly promoting its potential in wide applications. Demos are available at https://taoranyi.com/gaussiandreamerpro/.