ViBiDSampler：利用双向扩散采样器增强视频插值

ViBiDSampler: Enhancing Video Interpolation Using Bidirectional Diffusion Sampler

October 8, 2024

作者: Serin Yang, Taesung Kwon, Jong Chul Ye

cs.AI

摘要

最近在大规模文本到视频（T2V）和图像到视频（I2V）扩散模型方面取得的进展极大地增强了视频生成，特别是在关键帧插值方面。然而，当前的图像到视频扩散模型虽然在从单个条件帧生成视频方面很强大，但需要针对两帧（起始和结束）条件生成进行调整，这对于有效的有界插值至关重要。不幸的是，现有的将时间正向和反向路径并行融合的方法经常出现离开流形的问题，导致伪影或需要多次迭代重新加噪。在这项工作中，我们引入了一种新颖的双向采样策略，以解决这些离开流形的问题，而无需进行大量重新加噪或微调。我们的方法沿着正向和反向路径进行顺序采样，分别以起始和结束帧为条件，确保更连贯和在流形上生成中间帧。此外，我们还结合了先进的引导技术CFG++和DDS，进一步增强插值过程。通过整合这些技术，我们的方法实现了最先进的性能，在关键帧之间高效生成高质量、流畅的视频。在单个3090 GPU上，我们的方法可以在仅195秒内以1024 x 576分辨率插值25帧，使其成为关键帧插值的领先解决方案。

English

Recent progress in large-scale text-to-video (T2V) and image-to-video (I2V) diffusion models has greatly enhanced video generation, especially in terms of keyframe interpolation. However, current image-to-video diffusion models, while powerful in generating videos from a single conditioning frame, need adaptation for two-frame (start & end) conditioned generation, which is essential for effective bounded interpolation. Unfortunately, existing approaches that fuse temporally forward and backward paths in parallel often suffer from off-manifold issues, leading to artifacts or requiring multiple iterative re-noising steps. In this work, we introduce a novel, bidirectional sampling strategy to address these off-manifold issues without requiring extensive re-noising or fine-tuning. Our method employs sequential sampling along both forward and backward paths, conditioned on the start and end frames, respectively, ensuring more coherent and on-manifold generation of intermediate frames. Additionally, we incorporate advanced guidance techniques, CFG++ and DDS, to further enhance the interpolation process. By integrating these, our method achieves state-of-the-art performance, efficiently generating high-quality, smooth videos between keyframes. On a single 3090 GPU, our method can interpolate 25 frames at 1024 x 576 resolution in just 195 seconds, establishing it as a leading solution for keyframe interpolation.