LightIt：扩散模型的照明建模与控制

LightIt: Illumination Modeling and Control for Diffusion Models

March 15, 2024

作者: Peter Kocsis, Julien Philip, Kalyan Sunkavalli, Matthias Nießner, Yannick Hold-Geoffroy

cs.AI

摘要

我们介绍了LightIt，这是一种用于图像生成的显式照明控制方法。最近的生成方法缺乏照明控制，而这对于图像生成的许多艺术方面至关重要，比如设定整体情绪或影视外观。为了克服这些限制，我们建议将生成条件设置为阴影和法线图。我们使用单次反射阴影来建模照明，其中包括投射阴影。我们首先训练一个阴影估计模块，生成一个真实世界图像和阴影对的数据集。然后，我们使用估计的阴影和法线作为输入来训练一个控制网络。我们的方法展示了在许多场景中高质量的图像生成和照明控制。此外，我们使用我们生成的数据集来训练一个保持身份的照明重定向模型，以图像和目标阴影为条件。我们的方法是第一个能够生成具有可控、一致照明的图像，并且与专门的照明重定向最先进方法表现一致。

English

We introduce LightIt, a method for explicit illumination control for image generation. Recent generative methods lack lighting control, which is crucial to numerous artistic aspects of image generation such as setting the overall mood or cinematic appearance. To overcome these limitations, we propose to condition the generation on shading and normal maps. We model the lighting with single bounce shading, which includes cast shadows. We first train a shading estimation module to generate a dataset of real-world images and shading pairs. Then, we train a control network using the estimated shading and normals as input. Our method demonstrates high-quality image generation and lighting control in numerous scenes. Additionally, we use our generated dataset to train an identity-preserving relighting model, conditioned on an image and a target shading. Our method is the first that enables the generation of images with controllable, consistent lighting and performs on par with specialized relighting state-of-the-art methods.