任何場景中的任何物件：逼真的影片物件插入

Anything in Any Scene: Photorealistic Video Object Insertion

January 30, 2024

作者: Chen Bai, Zeman Shao, Guoxiang Zhang, Di Liang, Jie Yang, Zhuorui Zhang, Yujian Guo, Chengzhang Zhong, Yiqiao Qiu, Zhendong Wang, Yichen Guan, Xiaoyin Zheng, Tao Wang, Cheng Lu

cs.AI

摘要

逼真的影片模擬已在各種應用中展現顯著潛力，從虛擬實境到電影製作皆然。尤其在捕捉現實世界影片場景不切實際或昂貴的情況下，其效果更為明顯。現有的影片模擬方法常常無法準確模擬光線環境、呈現物體幾何形狀，或實現高度逼真感。本文提出了「任何場景中的任何物件」，一個新穎且通用的逼真影片模擬框架，可將任何物件無縫地插入現有動態影片，並強調物理逼真感。我們提出的通用框架包含三個關鍵過程：1)將逼真物件整合到給定場景影片中，確保幾何逼真感的適當放置；2)估計天空和環境光照分佈，並模擬逼真陰影以增強光線逼真感；3)應用風格轉換網絡，精煉最終影片輸出以極大化逼真感。我們實驗性地證明「任何場景中的任何物件」框架可生成具有出色幾何逼真感、光線逼真感和逼真感的模擬影片。通過顯著減輕與影片數據生成相關的挑戰，我們的框架為獲取高質量影片提供了高效且具成本效益的解決方案。此外，其應用範圍遠不僅限於影片數據增強，在虛擬實境、影片編輯和各種其他以影片為中心的應用中展現出有前途的潛力。請查看我們的項目網站https://anythinginanyscene.github.io，以訪問我們的項目代碼和更多高分辨率影片結果。

English

Realistic video simulation has shown significant potential across diverse applications, from virtual reality to film production. This is particularly true for scenarios where capturing videos in real-world settings is either impractical or expensive. Existing approaches in video simulation often fail to accurately model the lighting environment, represent the object geometry, or achieve high levels of photorealism. In this paper, we propose Anything in Any Scene, a novel and generic framework for realistic video simulation that seamlessly inserts any object into an existing dynamic video with a strong emphasis on physical realism. Our proposed general framework encompasses three key processes: 1) integrating a realistic object into a given scene video with proper placement to ensure geometric realism; 2) estimating the sky and environmental lighting distribution and simulating realistic shadows to enhance the light realism; 3) employing a style transfer network that refines the final video output to maximize photorealism. We experimentally demonstrate that Anything in Any Scene framework produces simulated videos of great geometric realism, lighting realism, and photorealism. By significantly mitigating the challenges associated with video data generation, our framework offers an efficient and cost-effective solution for acquiring high-quality videos. Furthermore, its applications extend well beyond video data augmentation, showing promising potential in virtual reality, video editing, and various other video-centric applications. Please check our project website https://anythinginanyscene.github.io for access to our project code and more high-resolution video results.