一个物体价值64x64像素：通过图像扩散生成3D物体

An Object is Worth 64x64 Pixels: Generating 3D Object via Image Diffusion

August 6, 2024

作者: Xingguang Yan, Han-Hung Lee, Ziyu Wan, Angel X. Chang

cs.AI

摘要

我们提出了一种新方法，通过一种称为“物体图像”的表示来生成带有UV贴图的逼真3D模型。这种方法将表面几何、外观和补丁结构封装在一个64x64像素的图像中，有效地将复杂的3D形状转换为更易处理的2D格式。通过这种方式，我们解决了多边形网格中固有的几何和语义不规则性所带来的挑战。这种方法使我们能够直接将图像生成模型（如扩散变换器）用于3D形状生成。在ABO数据集上评估时，我们生成的带有补丁结构的形状实现了与最近的3D生成模型相媲美的点云FID，同时自然地支持PBR材质生成。

English

We introduce a new approach for generating realistic 3D models with UV maps through a representation termed "Object Images." This approach encapsulates surface geometry, appearance, and patch structures within a 64x64 pixel image, effectively converting complex 3D shapes into a more manageable 2D format. By doing so, we address the challenges of both geometric and semantic irregularity inherent in polygonal meshes. This method allows us to use image generation models, such as Diffusion Transformers, directly for 3D shape generation. Evaluated on the ABO dataset, our generated shapes with patch structures achieve point cloud FID comparable to recent 3D generative models, while naturally supporting PBR material generation.