AVTok：面向整体音频-视频生成的一维统一令牌化

摘要

音频-视频生成近期获得了前所未有的研究关注，旨在合成高质量的含音视频内容，实现听觉与视觉成分之间的细粒度同步和语义对齐。此前的方法主要采用双分支设计，为每种模态配备独立的标记化和生成模块，忽视了表征差距，且需要大量的计算资源进行适当训练。受一维视觉标记化最新进展的启发，我们提出了AVTok，这是一种专为整体音频-视频生成设计的新型统一标记器。AVTok采用基于双流Transformer的架构，配备共享的编码器-解码器和模态特定的可学习查询，能够高效且有效地将音频-视频对编码为紧凑的一维潜表征，并采用统一的码本。为应对阻碍AVTok利用对齐的音频-视觉信息的异质信息不平衡问题，我们设计了一种分层训练策略，以逐步实现每个模态的重建能力。大量实验表明，AVTok在音频-视频重建以及集成到下游管道（如音频到视频、视频到音频以及类别条件联合音频-视频生成）中均表现出色。AVTok为联合音频-视频标记化的挑战铺平了道路，并为构建用于音频-视频生成的统一大规模多模态模型提供了潜在方向。

English

Audio-video generation has recently gained unprecedented research attention, aiming to synthesize high-quality sounding video content with fine-grained synchronization and semantic alignment between the auditory and visual components. The preceding methods predominantly adopt a dual-branch design with separate tokenization and generation modules per modality, neglecting the representation gap while necessitating intensive computational resources for proper training. Inspired by recent advancements in one-dimensional visual tokenization, we present AVTok, a novel unified tokenizer designated for holistic audio-video generation. AVTok features a dual-stream transformer-based architecture with shared encoder-decoder and modal-specific learnable queries to efficiently and effectively encode an audio-video pair into a compact one-dimensional latent representation with a unified codebook. To cope with the heterogeneous information imbalance that hinders AVTok from exploiting aligned audio-visual information, we devise a hierarchical training strategy to progressively realize reconstruction capabilities for each modality. Extensive experiments demonstrate that AVTok excels both in audio-video reconstruction and when integrated into downstream pipelines for audio-to-video, video-to-audio, and class-conditional joint audio-video generation. AVTok paves the way for the challenge of joint audio-video tokenization and provides a potential direction to build unified large multimodal models for audio-video generation.