Relightify：透過擴散模型從單張圖像生成可重新照明的3D人臉

摘要

在擴散模型在圖像生成上取得顯著成功後，最近的研究也展示了它們在無監督方式下解決多個反問題的印象深刻能力，通過根據條件輸入適當限制取樣過程。受此啟發，本文提出了第一種利用擴散模型作為高度準確的三維臉部BRDF復原的先驗方法，僅從單張圖像開始。我們首先利用高質量的臉部反射UV數據集（漫反射和鏡面反照率以及法線），在不同照明設置下渲染以模擬自然RGB紋理，然後在渲染紋理和反射成分的串聯對上訓練無條件擴散模型。在測試時，我們對給定圖像擬合三維可變模型，並在部分UV紋理中展開臉部。通過從擴散模型取樣，同時保留觀察到的紋理部分完整，模型不僅填補了自遮蔽區域，還填補了未知的反射成分，在一個序列的去噪步驟中。與現有方法相比，我們直接從輸入圖像獲取觀察到的紋理，因此，結果更忠實和一致的反射估計。通過一系列定性和定量比較，我們展示了在紋理完成和反射重建任務中的優越性能。

English

Following the remarkable success of diffusion models on image generation, recent works have also demonstrated their impressive ability to address a number of inverse problems in an unsupervised way, by properly constraining the sampling process based on a conditioning input. Motivated by this, in this paper, we present the first approach to use diffusion models as a prior for highly accurate 3D facial BRDF reconstruction from a single image. We start by leveraging a high-quality UV dataset of facial reflectance (diffuse and specular albedo and normals), which we render under varying illumination settings to simulate natural RGB textures and, then, train an unconditional diffusion model on concatenated pairs of rendered textures and reflectance components. At test time, we fit a 3D morphable model to the given image and unwrap the face in a partial UV texture. By sampling from the diffusion model, while retaining the observed texture part intact, the model inpaints not only the self-occluded areas but also the unknown reflectance components, in a single sequence of denoising steps. In contrast to existing methods, we directly acquire the observed texture from the input image, thus, resulting in more faithful and consistent reflectance estimation. Through a series of qualitative and quantitative comparisons, we demonstrate superior performance in both texture completion as well as reflectance reconstruction tasks.

Relightify：透過擴散模型從單張圖像生成可重新照明的3D人臉

Relightify: Relightable 3D Faces from a Single Image via Diffusion Models

摘要

Support