BRDFusion：物理與生成結合的都市場景逆渲染

摘要

從拍攝的影片中對城市場景進行逆向渲染，使眾多應用成為可能，包括內容創作與自動駕駛模擬。基於物理的渲染方法能夠遵循並控制光照物理特性，但存在重建與渲染偽影。生成模型雖能產生逼真的影片，但其一致性和可控性有限。我們提出BRDFusion，一個結合兩種互補模型進行逆向與正向渲染的統一框架。具體而言，BRDFusion透過物理建模還原明確且一致的場景屬性，並利用生成先驗緩解最佳化中的模糊性。在正向渲染過程中，物理模型能根據場景配置提供可控渲染，而生成模型則負責去噪和修正偽影。因此，我們的方法能產出高品質影片，同時允許精確控制，在真實與合成場景中均優於基準方法。此外，BRDFusion支援新視角重打光、夜間模擬以及動態物體插入/編輯。專案頁面：https://shigon255.github.io/brdfusion-page/

English

Inverse rendering of urban scenes from captured videos enables numerous applications, including content creation and autonomous driving simulation. Physically-based rendering methods follow and control lighting physics, but suffer from reconstruction and rendering artifacts. While generative models produce realistic videos, they offer limited consistency and controllability. We present BRDFusion, a unified framework that combines two complementary models for inverse and forward rendering. Specifically, BRDFusion recovers explicit, consistent scene properties with physical modeling and alleviates optimization ambiguity with generative priors. During forward rendering, the physical model provides controllable rendering from the scene configuration, and the generative model denoises and fixes artifacts. Therefore, our method produces high-quality videos while allowing precise control, outperforming baselines in real and synthetic scenes. Moreover, BRDFusion supports novel-view relighting, night simulation, and dynamic object insertion/editing. Project page: https://shigon255.github.io/brdfusion-page/