UniDream：统一扩散先验用于可照明的文本到三维生成

摘要

最近文本到3D生成技术的进展显著推动了将文本描述转换为富有想象力、几何形状良好且纹理精细的3D对象。尽管取得了这些进展，但一个普遍存在的限制是扩散或重建模型中使用RGB数据，这经常导致模型具有固有的光照和阴影效果，从而减弱其逼真度，从而限制了它们在需要准确重照能力的应用中的可用性。为了弥合这一差距，我们提出了UniDream，这是一个文本到3D生成框架，通过整合统一的扩散先验。我们的方法包括三个主要组成部分：（1）双阶段训练过程，获得反照率-法线对齐的多视角扩散和重建模型，（2）基于训练的重建和扩散模型，使用得分蒸馏样本（SDS）的渐进生成过程，生成几何形状和反照率纹理，（3）创新地应用SDS来完成基于稳定扩散模型的PBR生成，同时保持固定的反照率。广泛的评估表明，UniDream在生成具有更清晰反照率纹理、更平滑表面、增强逼真度和优越重照能力的3D对象方面超越了现有方法。

English

Recent advancements in text-to-3D generation technology have significantly advanced the conversion of textual descriptions into imaginative well-geometrical and finely textured 3D objects. Despite these developments, a prevalent limitation arises from the use of RGB data in diffusion or reconstruction models, which often results in models with inherent lighting and shadows effects that detract from their realism, thereby limiting their usability in applications that demand accurate relighting capabilities. To bridge this gap, we present UniDream, a text-to-3D generation framework by incorporating unified diffusion priors. Our approach consists of three main components: (1) a dual-phase training process to get albedo-normal aligned multi-view diffusion and reconstruction models, (2) a progressive generation procedure for geometry and albedo-textures based on Score Distillation Sample (SDS) using the trained reconstruction and diffusion models, and (3) an innovative application of SDS for finalizing PBR generation while keeping a fixed albedo based on Stable Diffusion model. Extensive evaluations demonstrate that UniDream surpasses existing methods in generating 3D objects with clearer albedo textures, smoother surfaces, enhanced realism, and superior relighting capabilities.

UniDream：统一扩散先验用于可照明的文本到三维生成

UniDream: Unifying Diffusion Priors for Relightable Text-to-3D Generation

摘要

Support