FATE：帧级音视频时间嵌入

摘要

当狗张开嘴吠叫时，人类自然会识别这是什么声音以及它何时发生。构建具有相同能力的视听模型，需要能够同时捕捉语义和时间对齐的表征。现有方法在某一方面存在不足：嵌入模型匹配语义但丢失时间信息；同步模型捕捉时间偏移但缺乏语义理解。为弥合这一差距，我们提出FATE，即帧级视听时间嵌入（Frame-level Audio-visual Temporal Embedding）。与先前将每个模态池化为单一嵌入并丢弃时间信息的嵌入模型不同，FATE保留帧级序列，将其在物理时间线上对齐，并在严格对齐的帧对上计算相似度。与仅输出偏移预测的同步模型不同，FATE将同步编码在可复用的嵌入空间中，通过结合跨视频语义对比学习和视频内时间对比学习的联合目标进行训练，以同时捕捉声音内容及其发生时间。在三个任务中，FATE在时间和语义检索上大幅超越最强基线，在零样本设置下的事件定位上与全监督方法相当，并作为生成评估指标取得了与人类判断的最佳相关性。源代码可在https://github.com/guankaisi/FATE获取。

English

When a dog opens its mouth and barks, humans naturally recognize what the sound is and when it occurs. Building audio-visual models with this same ability requires representations that capture both semantic and temporal alignment. Current approaches fall short on one side or the other: embedding models match semantic but lose temporal information; synchronization models capture temporal offsets but lack semantic understanding. To bridge this gap, we propose FATE, Frame-level Audio-visual Temporal Embedding. Unlike prior embedding models that pool each modality into a single embedding and discard temporal information, FATE retains frame-level sequences, aligns them on the physical timeline, and computes similarity over strictly aligned frame pairs. Unlike synchronization models that output only an offset prediction, FATE encodes synchronization in a reusable embedding space, trained with a joint objective combining cross-video semantic and within-video temporal contrastive learning to capture both what sounds and when it occurs. Across three tasks, FATE surpasses the strongest baseline on temporal and semantic retrieval by a large margin, matches fully supervised methods on event localization in a zero-shot setting, and achieves the best correlation with human judgments as a generation evaluation metric. The source code can be found at https://github.com/guankaisi/FATE.