LightIt：擴散模型的照明建模與控制

LightIt: Illumination Modeling and Control for Diffusion Models

March 15, 2024

作者: Peter Kocsis, Julien Philip, Kalyan Sunkavalli, Matthias Nießner, Yannick Hold-Geoffroy

cs.AI

摘要

我們介紹了 LightIt，一種用於影像生成的明確照明控制方法。最近的生成方法缺乏照明控制，這對於影像生成的許多藝術方面至關重要，例如設定整體情緒或電影外觀。為了克服這些限制，我們建議將生成條件設定為陰影和法向圖。我們使用單次反射陰影來建模照明，其中包括投影陰影。我們首先訓練一個陰影估計模組來生成一組真實世界影像和陰影對。然後，我們使用估計的陰影和法向作為輸入來訓練一個控制網絡。我們的方法展示了高質量的影像生成和照明控制在許多場景中。此外，我們使用我們生成的數據集來訓練一個保持身份的照明模型，條件是一個影像和一個目標陰影。我們的方法是第一個能夠生成具有可控、一致照明的影像並與專門的照明最先進方法相媲美的方法。

English

We introduce LightIt, a method for explicit illumination control for image generation. Recent generative methods lack lighting control, which is crucial to numerous artistic aspects of image generation such as setting the overall mood or cinematic appearance. To overcome these limitations, we propose to condition the generation on shading and normal maps. We model the lighting with single bounce shading, which includes cast shadows. We first train a shading estimation module to generate a dataset of real-world images and shading pairs. Then, we train a control network using the estimated shading and normals as input. Our method demonstrates high-quality image generation and lighting control in numerous scenes. Additionally, we use our generated dataset to train an identity-preserving relighting model, conditioned on an image and a target shading. Our method is the first that enables the generation of images with controllable, consistent lighting and performs on par with specialized relighting state-of-the-art methods.