将“Chirp”与“Chat”分开：自监督的声音和语言视觉定位

Separating the "Chirp" from the "Chat": Self-supervised Visual Grounding of Sound and Language

June 9, 2024

作者: Mark Hamilton, Andrew Zisserman, John R. Hershey, William T. Freeman

cs.AI

摘要

我们提出了DenseAV，这是一种新颖的双编码器接地架构，通过观看视频仅学习高分辨率、语义丰富且视听对齐的特征。我们展示了DenseAV能够在没有明确定位监督的情况下发现单词的“含义”和声音的“位置”。此外，它能够在没有监督的情况下自动发现并区分这两种关联类型。我们展示了DenseAV的定位能力来自一种新的多头特征聚合算子，该算子直接比较密集图像和音频表示以进行对比学习。相比之下，许多学习“全局”音频和视频表示的其他系统无法定位单词和声音。最后，我们提供了两个新数据集，以改进通过语音和声音提示的语义分割的评估。在这些数据集和其他数据集上，我们展示了DenseAV在语音和声音提示的语义分割方面明显优于先前的技术。DenseAV在跨模态检索方面的性能优于之前的最先进技术ImageBind，且参数使用不到一半。项目页面：https://aka.ms/denseav {https://aka.ms/denseav}

English

We present DenseAV, a novel dual encoder grounding architecture that learns high-resolution, semantically meaningful, and audio-visually aligned features solely through watching videos. We show that DenseAV can discover the ``meaning'' of words and the ``location'' of sounds without explicit localization supervision. Furthermore, it automatically discovers and distinguishes between these two types of associations without supervision. We show that DenseAV's localization abilities arise from a new multi-head feature aggregation operator that directly compares dense image and audio representations for contrastive learning. In contrast, many other systems that learn ``global'' audio and video representations cannot localize words and sound. Finally, we contribute two new datasets to improve the evaluation of AV representations through speech and sound prompted semantic segmentation. On these and other datasets we show DenseAV dramatically outperforms the prior art on speech and sound prompted semantic segmentation. DenseAV outperforms the previous state-of-the-art, ImageBind, on cross-modal retrieval using fewer than half of the parameters. Project Page: https://aka.ms/denseav{https://aka.ms/denseav}