SAM-MT：即時互動式多目標視訊分割

摘要

現代影片物件分割（VOS）涉及追蹤並分割使用者指定的目標。儘管近期方法在單目標場景中已展現卓越表現，將其擴展至多目標設定時，通常需為每個個別目標複製單目標處理流程，導致隨著目標數量增加，畫面更新率（FPS）下降且延遲無上限。基於分割一切 2（SAM2），我們提出 SAM-MT，透過將模型轉化為即時多目標影片分割的互動式框架來解決此問題。SAM-MT 使用顯式查詢來表示不同個別目標，並同時以共享表徵處理全域上下文。它採用解耦遮罩注意力機制，以維持個別身份不受跨目標干擾，並利用稀疏記憶體實現穩定的時序演進，同時針對遮擋處理與重疊預防設計專門策略。SAM-MT 成功將延遲與目標數量脫鉤，實現與單目標基準相當的即時速度（對 10 個目標達 >36 FPS），同時維持 SAM2 穩健的影片分割表現。

English

Modern Video Object Segmentation (VOS) involves tracking and segmenting user-specified targets. While recent approaches have achieved remarkable performance in single-target scenarios, extending them to multi-target settings typically involves replicating the single-target processing for each individual object, resulting in reduced frame rates (FPS) with unbounded latency as target count increases. Built upon Segment Anything 2 (SAM2), we propose SAM-MT, which addresses this by transforming the model into an interactive framework for real-time Multi-Target video segmentation. SAM-MT uses explicit queries to represent different individual targets, in parallel with a shared representation for global context. It employs decoupled masked attention to keep individual identities distinct from cross-target interference, and sparse memory for stable temporal evolution, along with specialized strategies for occlusion handling and overlap prevention. SAM-MT successfully decouples latency from the number of targets, achieving real-time speed on par with single-target baselines (>36 FPS for 10 targets) while maintaining SAM2's robust video segmentation performance.