AnimateDiff：动画您的个性化文本到图像扩散模型，无需特定调整。

摘要

随着文本到图像模型（例如，稳定扩散）以及相应的个性化技术（如DreamBooth和LoRA）的进步，每个人都可以以较低的成本将他们的想象力体现为高质量图像。随之而来的是对图像动画技术的巨大需求，以进一步将生成的静态图像与运动动态相结合。在本报告中，我们提出了一个实用框架，用于一劳永逸地为大多数现有的个性化文本到图像模型添加动画效果，节省了针对特定模型的调整工作。所提出的框架的核心是将一个新初始化的运动建模模块插入冻结的文本到图像模型中，并在视频剪辑上对其进行训练，以提炼合理的运动先验知识。一旦训练完成，通过简单地注入这个运动建模模块，所有从相同基础T2I衍生的个性化版本都会立即成为由文本驱动的模型，产生多样化和个性化的动画图像。我们对跨动漫图片和逼真照片领域的几个公共代表性个性化文本到图像模型进行评估，并展示了我们提出的框架如何帮助这些模型生成在时间上平滑的动画片段，同时保留其输出的领域和多样性。代码和预训练权重将在https://animatediff.github.io/ 上公开提供。

English

With the advance of text-to-image models (e.g., Stable Diffusion) and corresponding personalization techniques such as DreamBooth and LoRA, everyone can manifest their imagination into high-quality images at an affordable cost. Subsequently, there is a great demand for image animation techniques to further combine generated static images with motion dynamics. In this report, we propose a practical framework to animate most of the existing personalized text-to-image models once and for all, saving efforts in model-specific tuning. At the core of the proposed framework is to insert a newly initialized motion modeling module into the frozen text-to-image model and train it on video clips to distill reasonable motion priors. Once trained, by simply injecting this motion modeling module, all personalized versions derived from the same base T2I readily become text-driven models that produce diverse and personalized animated images. We conduct our evaluation on several public representative personalized text-to-image models across anime pictures and realistic photographs, and demonstrate that our proposed framework helps these models generate temporally smooth animation clips while preserving the domain and diversity of their outputs. Code and pre-trained weights will be publicly available at https://animatediff.github.io/ .

AnimateDiff：动画您的个性化文本到图像扩散模型，无需特定调整。

AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning

摘要

Support