局部注意力SDF擴散用於可控3D形狀生成

摘要

儘管最近3D生成神經網絡的快速演進顯著改善了3D形狀生成，但對於普通用戶來說，創建3D形狀並控制生成形狀的局部幾何仍不方便。為應對這些挑戰，我們提出了一個基於擴散的3D生成框架--局部關注SDF擴散，以模擬合理的3D形狀，通過2D草圖圖像輸入。我們的方法建立在一個兩階段擴散模型上。第一階段稱為佔用擴散，旨在生成一個低分辨率佔用場，以逼近形狀外殼。第二階段稱為SDF擴散，合成一個高分辨率的符號距離場，在第一階段確定的佔用體素內提取精細幾何。我們的模型借助一種新穎的視角感知局部關注機制進行圖像條件下的形狀生成，利用2D圖像補丁特徵引導3D體素特徵學習，極大地提高了局部可控性和模型泛化能力。通過在草圖條件和類別條件下進行的大量實驗，我們驗證並展示了我們的方法提供合理和多樣的3D形狀的能力，以及相對於現有工作的優越可控性和泛化能力。我們的代碼和訓練模型可在以下鏈接找到：https://zhengxinyang.github.io/projects/LAS-Diffusion.html

English

Although the recent rapid evolution of 3D generative neural networks greatly improves 3D shape generation, it is still not convenient for ordinary users to create 3D shapes and control the local geometry of generated shapes. To address these challenges, we propose a diffusion-based 3D generation framework -- locally attentional SDF diffusion, to model plausible 3D shapes, via 2D sketch image input. Our method is built on a two-stage diffusion model. The first stage, named occupancy-diffusion, aims to generate a low-resolution occupancy field to approximate the shape shell. The second stage, named SDF-diffusion, synthesizes a high-resolution signed distance field within the occupied voxels determined by the first stage to extract fine geometry. Our model is empowered by a novel view-aware local attention mechanism for image-conditioned shape generation, which takes advantage of 2D image patch features to guide 3D voxel feature learning, greatly improving local controllability and model generalizability. Through extensive experiments in sketch-conditioned and category-conditioned 3D shape generation tasks, we validate and demonstrate the ability of our method to provide plausible and diverse 3D shapes, as well as its superior controllability and generalizability over existing work. Our code and trained models are available at https://zhengxinyang.github.io/projects/LAS-Diffusion.html

局部注意力SDF擴散用於可控3D形狀生成

Locally Attentional SDF Diffusion for Controllable 3D Shape Generation

摘要

Support