VideoPrism：用於視頻理解的基礎視覺編碼器

摘要

我們介紹了VideoPrism，一種通用的影片編碼器，可使用單一凍結模型應對多樣的影片理解任務。我們在一個包含3600萬高質量影片標題對和5.82億影片片段的異質語料庫上對VideoPrism進行預訓練，其中包含帶有噪聲平行文本（例如ASR轉錄）的影片片段。預訓練方法改進了遮罩自編碼，通過全局-局部蒸餾語義影片嵌入和標記洗牌方案，使VideoPrism能夠主要專注於影片模態，同時利用與影片相關的寶貴文本。我們在四個廣泛的影片理解任務組上對VideoPrism進行了廣泛測試，從網絡影片問答到科學CV，並在33個影片理解基準測試中的30個上實現了最先進的性能。

English

We introduce VideoPrism, a general-purpose video encoder that tackles diverse video understanding tasks with a single frozen model. We pretrain VideoPrism on a heterogeneous corpus containing 36M high-quality video-caption pairs and 582M video clips with noisy parallel text (e.g., ASR transcripts). The pretraining approach improves upon masked autoencoding by global-local distillation of semantic video embeddings and a token shuffling scheme, enabling VideoPrism to focus primarily on the video modality while leveraging the invaluable text associated with videos. We extensively test VideoPrism on four broad groups of video understanding tasks, from web video question answering to CV for science, achieving state-of-the-art performance on 30 out of 33 video understanding benchmarks.

VideoPrism：用於視頻理解的基礎視覺編碼器

VideoPrism: A Foundational Visual Encoder for Video Understanding

摘要

Support