Kalman启发的特征传播用于视频人脸超分辨率

Kalman-Inspired Feature Propagation for Video Face Super-Resolution

August 9, 2024

作者: Ruicheng Feng, Chongyi Li, Chen Change Loy

cs.AI

摘要

尽管人脸图像超分辨率取得了令人期待的进展，但视频人脸超分辨率仍相对未被充分探索。现有方法要么将通用视频超分辨率网络调整为人脸数据集，要么独立地将已建立的人脸图像超分辨率模型应用于各个视频帧。这些范式在重建面部细节或保持时间一致性方面都面临挑战。为解决这些问题，我们引入了一种名为Kalman启发特征传播（KEEP）的新框架，旨在随时间保持稳定的人脸先验。Kalman滤波原理赋予我们的方法一种循环能力，利用先前恢复的帧的信息来指导和调节当前帧的恢复过程。大量实验证明了我们的方法在跨视频帧一致捕获面部细节方面的有效性。代码和视频演示可在https://jnjaby.github.io/projects/KEEP找到。

English

Despite the promising progress of face image super-resolution, video face super-resolution remains relatively under-explored. Existing approaches either adapt general video super-resolution networks to face datasets or apply established face image super-resolution models independently on individual video frames. These paradigms encounter challenges either in reconstructing facial details or maintaining temporal consistency. To address these issues, we introduce a novel framework called Kalman-inspired Feature Propagation (KEEP), designed to maintain a stable face prior over time. The Kalman filtering principles offer our method a recurrent ability to use the information from previously restored frames to guide and regulate the restoration process of the current frame. Extensive experiments demonstrate the effectiveness of our method in capturing facial details consistently across video frames. Code and video demo are available at https://jnjaby.github.io/projects/KEEP.