一段视频价值4096个标记：通过口头表达故事视频来理解它们在零样本情况下。

摘要

多媒体内容，如广告和故事视频，展示了丰富的创造力和多种形式。它们融合了文本、视觉、音频和叙事技巧等元素，运用情感、象征和口号等手段来传达意义。虽然先前在多媒体理解方面的研究主要集中在具有特定动作的视频，比如烹饪，但缺乏大规模注释的训练数据集，阻碍了为现实应用开发性能令人满意的监督学习模型。然而，大型语言模型（LLMs）的兴起在各种自然语言处理（NLP）任务中见证了显著的零-shot性能，如情感分类、问答和主题分类。为了弥合多媒体理解中的性能差距，我们提出了通过用自然语言生成故事视频描述，然后在生成的故事上执行视频理解任务，而不是在原始视频上执行。通过对五项视频理解任务进行大量实验，我们证明了我们的方法，尽管是零-shot，但在视频理解方面取得了比监督基线显著更好的结果。此外，为了缓解故事理解基准的不足，我们公开发布了计算社会科学中一项关键任务的第一个数据集，即说服策略识别。

English

Multimedia content, such as advertisements and story videos, exhibit a rich blend of creativity and multiple modalities. They incorporate elements like text, visuals, audio, and storytelling techniques, employing devices like emotions, symbolism, and slogans to convey meaning. While previous research in multimedia understanding has focused mainly on videos with specific actions like cooking, there is a dearth of large annotated training datasets, hindering the development of supervised learning models with satisfactory performance for real-world applications. However, the rise of large language models (LLMs) has witnessed remarkable zero-shot performance in various natural language processing (NLP) tasks, such as emotion classification, question-answering, and topic classification. To bridge this performance gap in multimedia understanding, we propose verbalizing story videos to generate their descriptions in natural language and then performing video-understanding tasks on the generated story as opposed to the original video. Through extensive experiments on five video-understanding tasks, we demonstrate that our method, despite being zero-shot, achieves significantly better results than supervised baselines for video understanding. Further, alleviating a lack of story understanding benchmarks, we publicly release the first dataset on a crucial task in computational social science, persuasion strategy identification.

一段视频价值4096个标记：通过口头表达故事视频来理解它们在零样本情况下。

A Video Is Worth 4096 Tokens: Verbalize Story Videos To Understand Them In Zero Shot

摘要

Support