框架：交互式帧插值

Framer: Interactive Frame Interpolation

October 24, 2024

作者: Wen Wang, Qiuyu Wang, Kecheng Zheng, Hao Ouyang, Zhekai Chen, Biao Gong, Hao Chen, Yujun Shen, Chunhua Shen

cs.AI

摘要

我们提出了Framer，用于交互式帧插值，旨在根据用户的创意产生在两幅图像之间平滑过渡的帧。具体而言，除了接受起始帧和结束帧作为输入外，我们的方法还支持通过调整一些选定关键点的轨迹来定制过渡过程。这种设计有两个明显的好处。首先，融入人类交互可以缓解由于将一幅图像转换为另一幅图像的众多可能性而产生的问题，从而实现对局部运动的更精细控制。其次，作为最基本的交互形式，关键点有助于建立跨帧的对应关系，增强模型处理具有挑战性的情况的能力（例如，起始帧和结束帧上的对象具有不同的形状和风格）。值得注意的是，我们的系统还提供了“自动驾驶”模式，其中我们引入一个模块来自动估计关键点并优化轨迹，以简化实际使用。大量实验结果展示了Framer在各种应用中的出色性能，例如图像变形、延时视频生成、卡通插值等。代码、模型和界面将会发布以促进进一步研究。

English

We propose Framer for interactive frame interpolation, which targets producing smoothly transitioning frames between two images as per user creativity. Concretely, besides taking the start and end frames as inputs, our approach supports customizing the transition process by tailoring the trajectory of some selected keypoints. Such a design enjoys two clear benefits. First, incorporating human interaction mitigates the issue arising from numerous possibilities of transforming one image to another, and in turn enables finer control of local motions. Second, as the most basic form of interaction, keypoints help establish the correspondence across frames, enhancing the model to handle challenging cases (e.g., objects on the start and end frames are of different shapes and styles). It is noteworthy that our system also offers an "autopilot" mode, where we introduce a module to estimate the keypoints and refine the trajectory automatically, to simplify the usage in practice. Extensive experimental results demonstrate the appealing performance of Framer on various applications, such as image morphing, time-lapse video generation, cartoon interpolation, etc. The code, the model, and the interface will be released to facilitate further research.