ChatPaper.aiChatPaper

Control4D:通過從2D擴散式編輯器學習4D GAN的動態肖像編輯

Control4D: Dynamic Portrait Editing by Learning 4D GAN from 2D Diffusion-based Editor

May 31, 2023
作者: Ruizhi Shao, Jingxiang Sun, Cheng Peng, Zerong Zheng, Boyao Zhou, Hongwen Zhang, Yebin Liu
cs.AI

摘要

近年來,在使用文字指令編輯圖像方面取得了相當大的成就。當將這些編輯器應用於動態場景編輯時,由於這些2D編輯器的逐幀特性,新風格場景往往在時間上不一致。為了應對這個問題,我們提出了Control4D,這是一種新穎的方法,用於高保真度和時間一致性的4D肖像編輯。Control4D基於一種高效的4D表示形式,並配備了一個2D基於擴散的編輯器。我們的方法不是直接從編輯器中獲取監督,而是從中學習一個4D GAN,避免不一致的監督信號。具體來說,我們利用鑑別器來根據編輯後的圖像學習生成分佈,然後用鑑別信號來更新生成器。為了實現更穩定的訓練,從編輯後的圖像中提取多層信息,並用於促進生成器的學習。實驗結果顯示,Control4D超越了先前的方法,實現了更具照片逼真度和一致性的4D編輯性能。我們項目網站的鏈接為https://control4darxiv.github.io。
English
Recent years have witnessed considerable achievements in editing images with text instructions. When applying these editors to dynamic scene editing, the new-style scene tends to be temporally inconsistent due to the frame-by-frame nature of these 2D editors. To tackle this issue, we propose Control4D, a novel approach for high-fidelity and temporally consistent 4D portrait editing. Control4D is built upon an efficient 4D representation with a 2D diffusion-based editor. Instead of using direct supervisions from the editor, our method learns a 4D GAN from it and avoids the inconsistent supervision signals. Specifically, we employ a discriminator to learn the generation distribution based on the edited images and then update the generator with the discrimination signals. For more stable training, multi-level information is extracted from the edited images and used to facilitate the learning of the generator. Experimental results show that Control4D surpasses previous approaches and achieves more photo-realistic and consistent 4D editing performances. The link to our project website is https://control4darxiv.github.io.
PDF22December 15, 2024