神经 Gaffer：通过扩散重新点亮任何物体

摘要

单图像重照明是一项具有挑战性的任务，涉及对几何、材质和光照之间复杂相互作用的推理。许多先前的方法要么仅支持特定类别的图像，如肖像，要么需要特殊的拍摄条件，比如使用手电筒。另外，一些方法明确地将场景分解为固有组件，如法线和BRDF，但这可能不准确或表达不足。在这项工作中，我们提出了一种新颖的端到端二维重照明扩散模型，名为神经Gaffer，它可以接受任何对象的单个图像，并能在任何新颖的环境光照条件下合成准确、高质量的重照图像，只需将图像生成器置于目标环境图的条件下，而无需明确场景分解。我们的方法基于一个预训练的扩散模型，并在一个合成重照数据集上对其进行微调，揭示并利用扩散模型中存在的对光照的固有理解。我们在合成和野外互联网图像上评估了我们的模型，并展示了它在泛化和准确性方面的优势。此外，通过与其他生成方法结合，我们的模型使许多下游的二维任务成为可能，如基于文本的重照和对象插入。我们的模型还可以作为强大的重照先验，用于三维任务，比如重照辐射场。

English

Single-image relighting is a challenging task that involves reasoning about the complex interplay between geometry, materials, and lighting. Many prior methods either support only specific categories of images, such as portraits, or require special capture conditions, like using a flashlight. Alternatively, some methods explicitly decompose a scene into intrinsic components, such as normals and BRDFs, which can be inaccurate or under-expressive. In this work, we propose a novel end-to-end 2D relighting diffusion model, called Neural Gaffer, that takes a single image of any object and can synthesize an accurate, high-quality relit image under any novel environmental lighting condition, simply by conditioning an image generator on a target environment map, without an explicit scene decomposition. Our method builds on a pre-trained diffusion model, and fine-tunes it on a synthetic relighting dataset, revealing and harnessing the inherent understanding of lighting present in the diffusion model. We evaluate our model on both synthetic and in-the-wild Internet imagery and demonstrate its advantages in terms of generalization and accuracy. Moreover, by combining with other generative methods, our model enables many downstream 2D tasks, such as text-based relighting and object insertion. Our model can also operate as a strong relighting prior for 3D tasks, such as relighting a radiance field.

神经 Gaffer：通过扩散重新点亮任何物体

Neural Gaffer: Relighting Any Object via Diffusion

摘要

Summary

Support

Support