Control4D：通过从2D扩散式编辑器中学习4D GAN进行动态肖像编辑

Control4D: Dynamic Portrait Editing by Learning 4D GAN from 2D Diffusion-based Editor

May 31, 2023

作者: Ruizhi Shao, Jingxiang Sun, Cheng Peng, Zerong Zheng, Boyao Zhou, Hongwen Zhang, Yebin Liu

cs.AI

摘要

近年来，在使用文本指令编辑图像方面取得了相当大的成就。将这些编辑器应用于动态场景编辑时，由于这些2D编辑器是逐帧进行的，新风格场景往往在时间上不一致。为了解决这个问题，我们提出了Control4D，这是一种用于高保真和时间一致的4D肖像编辑的新方法。Control4D基于高效的4D表示构建，配合一个2D基于扩散的编辑器。我们的方法不是直接从编辑器中获得监督，而是从中学习一个4D GAN，并避免不一致的监督信号。具体来说，我们使用鉴别器来学习基于编辑图像的生成分布，然后用鉴别信号更新生成器。为了更稳定的训练，我们从编辑图像中提取多级信息，并用于促进生成器的学习。实验结果显示，Control4D超越了先前的方法，实现了更逼真和一致的4D编辑性能。我们项目网站的链接是https://control4darxiv.github.io。

English

Recent years have witnessed considerable achievements in editing images with text instructions. When applying these editors to dynamic scene editing, the new-style scene tends to be temporally inconsistent due to the frame-by-frame nature of these 2D editors. To tackle this issue, we propose Control4D, a novel approach for high-fidelity and temporally consistent 4D portrait editing. Control4D is built upon an efficient 4D representation with a 2D diffusion-based editor. Instead of using direct supervisions from the editor, our method learns a 4D GAN from it and avoids the inconsistent supervision signals. Specifically, we employ a discriminator to learn the generation distribution based on the edited images and then update the generator with the discrimination signals. For more stable training, multi-level information is extracted from the edited images and used to facilitate the learning of the generator. Experimental results show that Control4D surpasses previous approaches and achieves more photo-realistic and consistent 4D editing performances. The link to our project website is https://control4darxiv.github.io.