Relightify：通过扩散模型从单个图像生成可重新照明的3D人脸

摘要

在扩散模型在图像生成方面取得显著成功后，最近的研究还展示了它们在以无监督方式解决多个反问题方面的令人印象深刻能力，通过根据条件输入适当约束采样过程。受此启发，在本文中，我们提出了首个利用扩散模型作为高精度三维人脸BRDF重建的先验的方法。我们首先利用高质量的人脸反射UV数据集（漫反射和镜面反照率以及法线），在不同照明设置下渲染以模拟自然RGB纹理，然后在渲染纹理和反射分量的连接对上训练一个无条件扩散模型。在测试时，我们将3D可塑模型拟合到给定图像中，并在部分UV纹理中展开面部。通过从扩散模型中采样，同时保留观察到的纹理部分不变，该模型不仅对自遮挡区域进行修复，还对未知的反射分量进行修复，在单个去噪步骤序列中。与现有方法相比，我们直接从输入图像中获取观察到的纹理，从而导致更忠实和一致的反射估计。通过一系列定性和定量比较，我们展示了在纹理完成和反射重建任务中的卓越性能。

English

Following the remarkable success of diffusion models on image generation, recent works have also demonstrated their impressive ability to address a number of inverse problems in an unsupervised way, by properly constraining the sampling process based on a conditioning input. Motivated by this, in this paper, we present the first approach to use diffusion models as a prior for highly accurate 3D facial BRDF reconstruction from a single image. We start by leveraging a high-quality UV dataset of facial reflectance (diffuse and specular albedo and normals), which we render under varying illumination settings to simulate natural RGB textures and, then, train an unconditional diffusion model on concatenated pairs of rendered textures and reflectance components. At test time, we fit a 3D morphable model to the given image and unwrap the face in a partial UV texture. By sampling from the diffusion model, while retaining the observed texture part intact, the model inpaints not only the self-occluded areas but also the unknown reflectance components, in a single sequence of denoising steps. In contrast to existing methods, we directly acquire the observed texture from the input image, thus, resulting in more faithful and consistent reflectance estimation. Through a series of qualitative and quantitative comparisons, we demonstrate superior performance in both texture completion as well as reflectance reconstruction tasks.

Relightify：通过扩散模型从单个图像生成可重新照明的3D人脸

Relightify: Relightable 3D Faces from a Single Image via Diffusion Models

摘要

Support