BroadWay：在无需训练的情况下提升您的文本到视频生成模型

BroadWay: Boost Your Text-to-Video Generation Model in a Training-free Way

October 8, 2024

作者: Jiazi Bu, Pengyang Ling, Pan Zhang, Tong Wu, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Dahua Lin, Jiaqi Wang

cs.AI

摘要

最近，文本到视频（T2V）生成模型备受关注，为视觉创作提供了便利。尽管具有巨大潜力，生成的视频可能存在缺陷，包括结构不合理、时间不一致和缺乏动态，通常导致几乎静态的视频。在这项工作中，我们发现不同块之间时间注意力图的差异与时间不一致的发生之间存在相关性。此外，我们观察到时间注意力图中包含的能量与生成视频中运动幅度的大小直接相关。基于这些观察，我们提出了BroadWay，一种无需训练的方法，可提高文本到视频生成的质量，而无需引入额外参数、增加内存或采样时间。具体而言，BroadWay由两个主要组件组成：1）时间自我引导通过减少不同解码器块之间时间注意力图的差异，改善生成视频的结构合理性和时间一致性。2）基于傅立叶的运动增强通过放大地图的能量，增强运动的幅度和丰富度。大量实验证明，BroadWay显著提高了文本到视频生成的质量，且附加成本微乎其微。

English

The text-to-video (T2V) generation models, offering convenient visual creation, have recently garnered increasing attention. Despite their substantial potential, the generated videos may present artifacts, including structural implausibility, temporal inconsistency, and a lack of motion, often resulting in near-static video. In this work, we have identified a correlation between the disparity of temporal attention maps across different blocks and the occurrence of temporal inconsistencies. Additionally, we have observed that the energy contained within the temporal attention maps is directly related to the magnitude of motion amplitude in the generated videos. Based on these observations, we present BroadWay, a training-free method to improve the quality of text-to-video generation without introducing additional parameters, augmenting memory or sampling time. Specifically, BroadWay is composed of two principal components: 1) Temporal Self-Guidance improves the structural plausibility and temporal consistency of generated videos by reducing the disparity between the temporal attention maps across various decoder blocks. 2) Fourier-based Motion Enhancement enhances the magnitude and richness of motion by amplifying the energy of the map. Extensive experiments demonstrate that BroadWay significantly improves the quality of text-to-video generation with negligible additional cost.