FATE：幀級音頻-視覺時間嵌入

摘要

當狗張開嘴吠叫時，人類自然而然能辨識該聲音為何以及發生於何時。建構具備同等能力的音視頻模型，需要能同時捕捉語義與時間對齊的表示。現有方法總在某一面向有所欠缺：嵌入模型匹配語義但喪失時間資訊；同步模型捕捉時間偏移但缺乏語義理解。為彌補此差距，我們提出 FATE，幀級音視頻時間嵌入（Frame-level Audio-visual Temporal Embedding）。不同於先前將各模態池化為單一嵌入並丟棄時間資訊的嵌入模型，FATE 保留幀級序列，在物理時間軸上對其對齊，並於嚴格對齊的幀對上計算相似度。不同於僅輸出偏移預測的同步模型，FATE 將同步編碼於可重複使用的嵌入空間中，並以結合跨視頻語義與視頻內時間對比學習的聯合目標進行訓練，以同時捕捉聲音為何及發生於何時。在三個任務中，FATE 在時間與語義檢索上大幅超越最強基線，在零樣本設定下的事件定位上媲美完全監督方法，並作為生成評估指標時，達到與人工評判的最佳相關性。原始碼可在 https://github.com/guankaisi/FATE 取得。

English

When a dog opens its mouth and barks, humans naturally recognize what the sound is and when it occurs. Building audio-visual models with this same ability requires representations that capture both semantic and temporal alignment. Current approaches fall short on one side or the other: embedding models match semantic but lose temporal information; synchronization models capture temporal offsets but lack semantic understanding. To bridge this gap, we propose FATE, Frame-level Audio-visual Temporal Embedding. Unlike prior embedding models that pool each modality into a single embedding and discard temporal information, FATE retains frame-level sequences, aligns them on the physical timeline, and computes similarity over strictly aligned frame pairs. Unlike synchronization models that output only an offset prediction, FATE encodes synchronization in a reusable embedding space, trained with a joint objective combining cross-video semantic and within-video temporal contrastive learning to capture both what sounds and when it occurs. Across three tasks, FATE surpasses the strongest baseline on temporal and semantic retrieval by a large margin, matches fully supervised methods on event localization in a zero-shot setting, and achieves the best correlation with human judgments as a generation evaluation metric. The source code can be found at https://github.com/guankaisi/FATE.