局部注意力SDF扩散用于可控的3D形状生成

摘要

尽管最近快速发展的3D生成神经网络极大地改进了3D形状生成，但普通用户仍然不方便创建3D形状和控制生成形状的局部几何。为了解决这些挑战，我们提出了一种基于扩散的3D生成框架 -- 局部注意力SDF扩散，用于通过2D草图图像输入建模可信的3D形状。我们的方法建立在一个两阶段扩散模型上。第一阶段名为占用扩散，旨在生成一个低分辨率的占用场，以近似形状外壳。第二阶段名为SDF扩散，合成一个高分辨率的有符号距离场，用于提取第一阶段确定的占用体素内的细节几何。我们的模型采用了一种新颖的视图感知局部注意力机制，用于基于图像的形状生成，利用2D图像块特征来引导3D体素特征学习，极大地提高了局部可控性和模型的泛化能力。通过在基于草图和基于类别的3D形状生成任务中进行大量实验，我们验证并展示了我们的方法提供可信和多样的3D形状的能力，以及相对于现有工作的卓越可控性和泛化能力。我们的代码和训练模型可在以下网址找到：https://zhengxinyang.github.io/projects/LAS-Diffusion.html

English

Although the recent rapid evolution of 3D generative neural networks greatly improves 3D shape generation, it is still not convenient for ordinary users to create 3D shapes and control the local geometry of generated shapes. To address these challenges, we propose a diffusion-based 3D generation framework -- locally attentional SDF diffusion, to model plausible 3D shapes, via 2D sketch image input. Our method is built on a two-stage diffusion model. The first stage, named occupancy-diffusion, aims to generate a low-resolution occupancy field to approximate the shape shell. The second stage, named SDF-diffusion, synthesizes a high-resolution signed distance field within the occupied voxels determined by the first stage to extract fine geometry. Our model is empowered by a novel view-aware local attention mechanism for image-conditioned shape generation, which takes advantage of 2D image patch features to guide 3D voxel feature learning, greatly improving local controllability and model generalizability. Through extensive experiments in sketch-conditioned and category-conditioned 3D shape generation tasks, we validate and demonstrate the ability of our method to provide plausible and diverse 3D shapes, as well as its superior controllability and generalizability over existing work. Our code and trained models are available at https://zhengxinyang.github.io/projects/LAS-Diffusion.html

局部注意力SDF扩散用于可控的3D形状生成

Locally Attentional SDF Diffusion for Controllable 3D Shape Generation

摘要

Support