保留或丟棄？用於緊湊視頻表徵的自適應標記器

摘要

潛在擴散模型已成為高保真影像與視訊合成的主流框架，其透過變分自編碼器在緊湊的潛在空間中運作，以提升計算效率同時不損害視覺品質。然而，傳統的變分自編碼器對於視訊資料而言並非最佳選擇，因為它們採用固定的壓縮率，無法適應時空內容的複雜度變化。我們提出KATok（保留或丟棄？自適應緊湊視訊表徵的分詞器），這是一個基於Transformer的變分自編碼器，其結合了與潛在令牌共同學習的自適應令牌選擇器。透過評估每個令牌的內容豐富度作為保留或丟棄之機率，令牌選擇器能有效丟棄不具資訊性的令牌，自然實現資料依賴性的壓縮。將自適應分詞應用於擴散模型可能導致空間錯位，因為令牌丟棄可能擾亂原始的時空結構。為緩解此問題，我們提出兩種位置預測策略：級聯生成與聯合生成，以確保空間一致性。我們經由實驗證明，我們的模型在先進的壓縮率下實現了強勁的重建與生成品質。對視訊資料的進一步分析顯示，此改進主要來自於降低時空冗餘並移除不具資訊性的令牌，此結論獲得了量化與質性結果的共同支持。

English

Latent diffusion models have emerged as a dominant framework for high-fidelity image and video synthesis, operating in compact latent spaces with variational autoencoders (VAEs) to enhance computational efficiency without compromising visual quality. However, conventional VAEs are suboptimal for video data as they employ fixed compression ratios that cannot adapt to the varying complexity of spatio-temporal content. We present KATok (Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation), a transformer-based VAE that incorporates an adaptive token selector which is jointly learned with latent tokens. By evaluating each token's content-richness as keep-or-drop probability, the token selector effectively discards uninformative tokens, naturally allowing data-dependent compression. Applying adaptive tokenization to diffusion models may cause spatial misalignment, as token dropping can disturb the original spatio-temporal structure. To alleviate this issue, we propose two position-prediction strategies: cascaded and joint generation, to ensure spatial consistency. We empirically show that our model achieves strong reconstruction and generation quality at a state-of-the-art compression ratio. Further analysis on video data reveals that this improvement is primarily achieved by reducing spatio-temporal redundancy and removing uninformative tokens, as supported by both quantitative and qualitative results.