MovieLLM:利用人工智能生成的电影增强长视频理解
MovieLLM: Enhancing Long Video Understanding with AI-Generated Movies
March 3, 2024
作者: Zhende Song, Chenchen Wang, Jiamu Sheng, Chi Zhang, Gang Yu, Jiayuan Fan, Tao Chen
cs.AI
摘要
多模型的发展标志着机器理解视频迈出了重要的一步。这些模型已经显示出在分析短视频剪辑方面很有前景。然而,当涉及到像电影这样的更长格式时,它们经常表现不佳。主要障碍在于缺乏高质量、多样化的视频数据以及收集或标注此类数据所需的大量工作。面对这些挑战,我们提出了MovieLLM,这是一个旨在为长视频创建合成高质量数据的新颖框架。该框架利用了GPT-4和文本到图像模型的强大能力,生成详细的剧本和相应的视觉效果。我们的方法突出表现在其灵活性和可扩展性上,使其成为传统数据收集方法的优越选择。我们广泛的实验证实,由MovieLLM生成的数据显著提高了多模型在理解复杂视频叙事方面的性能,克服了现有数据集在稀缺性和偏见方面的局限性。
English
The development of multimodal models has marked a significant step forward in
how machines understand videos. These models have shown promise in analyzing
short video clips. However, when it comes to longer formats like movies, they
often fall short. The main hurdles are the lack of high-quality, diverse video
data and the intensive work required to collect or annotate such data. In the
face of these challenges, we propose MovieLLM, a novel framework designed to
create synthetic, high-quality data for long videos. This framework leverages
the power of GPT-4 and text-to-image models to generate detailed scripts and
corresponding visuals. Our approach stands out for its flexibility and
scalability, making it a superior alternative to traditional data collection
methods. Our extensive experiments validate that the data produced by MovieLLM
significantly improves the performance of multimodal models in understanding
complex video narratives, overcoming the limitations of existing datasets
regarding scarcity and bias.