保留还是丢弃？用于紧凑视频表示的自适应分词器

摘要

潜在扩散模型已成为高保真图像和视频合成的主流框架，其在紧凑的潜在空间中结合变分自编码器（VAEs）运行，以提升计算效率而不损害视觉质量。然而，传统VAE对于视频数据并非最优，因为它们采用固定压缩比，无法适应时空内容复杂度的变化。我们提出KATok（Keep-or-Drop？用于紧凑视频表示的自适应标记器），一种基于Transformer的VAE，其整合了一个与潜在标记联合学习的自适应标记选择器。通过将每个标记的内容丰富度评估为保留或丢弃的概率，标记选择器能够有效丢弃信息量低的标记，自然实现依赖于数据的压缩。将自适应标记化应用于扩散模型可能引发空间错位，因为标记丢弃会扰动原始的时空结构。为缓解该问题，我们提出两种位置预测策略：级联生成和联合生成，以确保空间一致性。实验表明，我们的模型在最先进的压缩比下实现了强大的重建与生成质量。对视频数据的进一步分析揭示，这一改进主要通过减少时空冗余和去除信息量低的标记而实现，定量与定性结果均支持该结论。

English

Latent diffusion models have emerged as a dominant framework for high-fidelity image and video synthesis, operating in compact latent spaces with variational autoencoders (VAEs) to enhance computational efficiency without compromising visual quality. However, conventional VAEs are suboptimal for video data as they employ fixed compression ratios that cannot adapt to the varying complexity of spatio-temporal content. We present KATok (Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation), a transformer-based VAE that incorporates an adaptive token selector which is jointly learned with latent tokens. By evaluating each token's content-richness as keep-or-drop probability, the token selector effectively discards uninformative tokens, naturally allowing data-dependent compression. Applying adaptive tokenization to diffusion models may cause spatial misalignment, as token dropping can disturb the original spatio-temporal structure. To alleviate this issue, we propose two position-prediction strategies: cascaded and joint generation, to ensure spatial consistency. We empirically show that our model achieves strong reconstruction and generation quality at a state-of-the-art compression ratio. Further analysis on video data reveals that this improvement is primarily achieved by reducing spatio-temporal redundancy and removing uninformative tokens, as supported by both quantitative and qualitative results.