LVCD：基于参考线稿的扩散模型视频上色

LVCD: Reference-based Lineart Video Colorization with Diffusion Models

September 19, 2024

作者: Zhitong Huang, Mohan Zhang, Jing Liao

cs.AI

摘要

我们提出了第一个基于参考线稿视频上色的视频扩散框架。与先前仅依赖图像生成模型逐帧上色线稿的方法不同，我们的方法利用大规模预训练的视频扩散模型来生成彩色动画视频。这种方法产生了更具时间一致性的结果，并且更适合处理大运动。首先，我们引入了Sketch-guided ControlNet，为图像到视频扩散模型提供额外控制，用于可控视频合成，实现了基于线稿生成动画视频。然后，我们提出了Reference Attention，以促进从参考帧向包含快速和广泛运动的其他帧传递颜色。最后，我们提出了一种新颖的顺序采样方案，结合了Overlapped Blending Module和Prev-Reference Attention，以超越原始固定长度限制，实现长视频上色。定性和定量结果表明，我们的方法在帧和视频质量以及时间一致性方面明显优于最先进的技术。此外，我们的方法能够生成具有大运动的高质量、长时间一致的动画视频，这是先前方法无法实现的。我们的代码和模型可在https://luckyhzt.github.io/lvcd获得。

English

We propose the first video diffusion framework for reference-based lineart video colorization. Unlike previous works that rely solely on image generative models to colorize lineart frame by frame, our approach leverages a large-scale pretrained video diffusion model to generate colorized animation videos. This approach leads to more temporally consistent results and is better equipped to handle large motions. Firstly, we introduce Sketch-guided ControlNet which provides additional control to finetune an image-to-video diffusion model for controllable video synthesis, enabling the generation of animation videos conditioned on lineart. We then propose Reference Attention to facilitate the transfer of colors from the reference frame to other frames containing fast and expansive motions. Finally, we present a novel scheme for sequential sampling, incorporating the Overlapped Blending Module and Prev-Reference Attention, to extend the video diffusion model beyond its original fixed-length limitation for long video colorization. Both qualitative and quantitative results demonstrate that our method significantly outperforms state-of-the-art techniques in terms of frame and video quality, as well as temporal consistency. Moreover, our method is capable of generating high-quality, long temporal-consistent animation videos with large motions, which is not achievable in previous works. Our code and model are available at https://luckyhzt.github.io/lvcd.