神經 Gaffer：透過擴散重新照亮任何物體

Neural Gaffer: Relighting Any Object via Diffusion

June 11, 2024

作者: Haian Jin, Yuan Li, Fujun Luan, Yuanbo Xiangli, Sai Bi, Kai Zhang, Zexiang Xu, Jin Sun, Noah Snavely

cs.AI

摘要

單張圖像燈光重製是一項具有挑戰性的任務，需要推理幾何、材質和燈光之間的複雜互動。許多先前的方法僅支持特定類別的圖像，如肖像，或需要特殊的拍攝條件，例如使用手電筒。另外，一些方法明確地將場景分解為內在組件，如法線和BRDF，但這可能不準確或表達不足。在這項工作中，我們提出了一種新穎的端對端2D燈光重製擴散模型，稱為神經Gaffer，它接受任何物體的單張圖像，可以在任何新環境燈光條件下合成準確、高質量的燈光重製圖像，只需將圖像生成器條件化為目標環境地圖，而無需明確場景分解。我們的方法基於預先訓練的擴散模型，並在合成燈光重製數據集上進行微調，揭示並利用擴散模型中存在的對燈光的固有理解。我們在合成和野外互聯網圖像上評估我們的模型，並展示其在泛化和準確性方面的優勢。此外，通過與其他生成方法結合，我們的模型使許多下游2D任務成為可能，如基於文本的燈光重製和物體插入。我們的模型還可以作為3D任務的強烈燈光先驗，例如對輻射場進行燈光重製。

English

Single-image relighting is a challenging task that involves reasoning about the complex interplay between geometry, materials, and lighting. Many prior methods either support only specific categories of images, such as portraits, or require special capture conditions, like using a flashlight. Alternatively, some methods explicitly decompose a scene into intrinsic components, such as normals and BRDFs, which can be inaccurate or under-expressive. In this work, we propose a novel end-to-end 2D relighting diffusion model, called Neural Gaffer, that takes a single image of any object and can synthesize an accurate, high-quality relit image under any novel environmental lighting condition, simply by conditioning an image generator on a target environment map, without an explicit scene decomposition. Our method builds on a pre-trained diffusion model, and fine-tunes it on a synthetic relighting dataset, revealing and harnessing the inherent understanding of lighting present in the diffusion model. We evaluate our model on both synthetic and in-the-wild Internet imagery and demonstrate its advantages in terms of generalization and accuracy. Moreover, by combining with other generative methods, our model enables many downstream 2D tasks, such as text-based relighting and object insertion. Our model can also operate as a strong relighting prior for 3D tasks, such as relighting a radiance field.