SemanTok：用于高效自回归视频生成的可预测语义标记

摘要

近期基于视频的世界模型将自回归（AR）预测的可扩展性与扩散模型的视觉质量结合起来。场景 tokenizer 的选择对于二者各自达到最优性能都至关重要，无论是在保真度还是语义方面。可变长度、从粗到细的 tokenizer 恰好能实现这一点：最初的粗粒度 token 承载片段的全局语义，而后续 token 则进一步明确细节。现有的灵活 tokenizer 仅在解码器早期隐藏状态上施加表示对齐（REPA）损失，而该目标转而可由解码器部分地通过其含噪输入满足。我们提出 SemanTok，一种灵活的视频 tokenizer：它将冻结的 DINO 特征输入其编码器，并加入轻量级头，仅从每个保留的 token 前缀中重建这些特征。SemanTok 在每个 AR 模型规模下都实现了高语义对齐和高视频保真度：一个 201M 的 SemanTok AR 模型与规模为其 3.4 倍的 VideoFlexTok AR 模型相当或更优，而更大的 SemanTok AR 模型进一步提升保真度。它在分布外类别上保持语义对齐，并在包括纯噪声在内的每个噪声水平上使解码器具有更高的语义对齐。它在重建和生成上均表现良好；其短 token 前缀预测成本更低，并能带来更好的生成保真度，而像素细节则留待后续 token 处理。

English

Recent video-based world models pair the scalability of autoregressive (AR) prediction with the visual quality of diffusion models. The choice of scene tokenizer is paramount for the optimal performance of each of these, both in terms of fidelity and semantics. Flexible-length, coarse-to-fine tokenizers yield exactly that: the first coarse tokens carry the clip's global semantics while later tokens further specify details. Existing flexible tokenizers only apply a representation-alignment (REPA) loss on early decoder hidden states, a target the decoder can partly meet from its noised input instead. We introduce SemanTok, a flexible video tokenizer that feeds frozen DINO features into its encoder and adds lightweight heads that reconstruct them from each retained token prefix alone. SemanTok achieves high semantic alignment and video fidelity at every AR model size: a 201M SemanTok AR model matches or beats a VideoFlexTok AR model 3.4times its size, and larger SemanTok AR models further improve fidelity. It keeps semantic alignment on out-of-distribution classes and gives the decoder higher semantic alignment at every noise level, including pure noise. It performs well in both reconstruction and generation, and its short token prefixes are cheaper to predict and give better generation fidelity, with pixel detail deferred to later tokens.