DreamVideo：使用定制主题和动作创作您的梦幻视频

摘要

使用扩散模型进行定制生成在图像生成方面取得了令人印象深刻的进展，但在具有挑战性的视频生成任务中仍然不尽人意，因为它需要对主体和动作进行可控性处理。为此，我们提出了DreamVideo，这是一种从所需主体的少量静态图像和目标运动的少量视频生成个性化视频的新方法。DreamVideo将这一任务分解为两个阶段，即主体学习和动作学习，通过利用预先训练的视频扩散模型。主体学习旨在准确捕捉所提供图像中主体的精细外观，这是通过结合文本反演和精心设计的身份适配器的微调来实现的。在动作学习中，我们设计了一个动作适配器，并在给定视频上进行微调，以有效地建模目标运动模式。结合这两个轻量级高效的适配器，可以灵活定制任何主体与任何动作。大量实验结果表明，我们的DreamVideo在定制视频生成方面优于最先进的方法。我们的项目页面位于https://dreamvideo-t2v.github.io。

English

Customized generation using diffusion models has made impressive progress in image generation, but remains unsatisfactory in the challenging video generation task, as it requires the controllability of both subjects and motions. To that end, we present DreamVideo, a novel approach to generating personalized videos from a few static images of the desired subject and a few videos of target motion. DreamVideo decouples this task into two stages, subject learning and motion learning, by leveraging a pre-trained video diffusion model. The subject learning aims to accurately capture the fine appearance of the subject from provided images, which is achieved by combining textual inversion and fine-tuning of our carefully designed identity adapter. In motion learning, we architect a motion adapter and fine-tune it on the given videos to effectively model the target motion pattern. Combining these two lightweight and efficient adapters allows for flexible customization of any subject with any motion. Extensive experimental results demonstrate the superior performance of our DreamVideo over the state-of-the-art methods for customized video generation. Our project page is at https://dreamvideo-t2v.github.io.

DreamVideo：使用定制主题和动作创作您的梦幻视频

DreamVideo: Composing Your Dream Videos with Customized Subject and Motion

摘要

Support