AVTok：面向整體音頻-視頻生成的一維統一標記化

摘要

音頻-視頻生成近期獲得了前所未有的研究關注，旨在合成高品質的有聲視頻內容，實現聽覺與視覺組件之間的細粒度同步與語義對齊。現有方法主要採用雙分支設計，為每種模態配備單獨的標記化與生成模組，未能解決表徵差距問題，同時需要大量運算資源進行適當訓練。受近期一維視覺標記化進展的啟發，我們提出AVTok，一種專為整體音頻-視頻生成設計的新型統一標記器。AVTok採用基於雙流變壓器架構的設計，配有共享編碼器-解碼器與模態特定可學習查詢，能高效且有效地將音頻-視頻對編碼為帶有統一碼本的緊湊一維潛在表徵。為應對阻礙AVTok利用對齊音頻-視覺資訊的異質資訊不平衡問題，我們設計了一套分層訓練策略，逐步實現各模態的重建能力。大量實驗表明，AVTok在音頻-視頻重建方面表現優異，且整合至下游流程（如音頻轉視頻、視頻轉音頻及類別條件聯合音頻-視頻生成）時亦效果顯著。AVTok為聯合音頻-視頻標記化挑戰鋪平了道路，並為構建用於音頻-視頻生成的統一大型多模態模型提供了潛在方向。

English

Audio-video generation has recently gained unprecedented research attention, aiming to synthesize high-quality sounding video content with fine-grained synchronization and semantic alignment between the auditory and visual components. The preceding methods predominantly adopt a dual-branch design with separate tokenization and generation modules per modality, neglecting the representation gap while necessitating intensive computational resources for proper training. Inspired by recent advancements in one-dimensional visual tokenization, we present AVTok, a novel unified tokenizer designated for holistic audio-video generation. AVTok features a dual-stream transformer-based architecture with shared encoder-decoder and modal-specific learnable queries to efficiently and effectively encode an audio-video pair into a compact one-dimensional latent representation with a unified codebook. To cope with the heterogeneous information imbalance that hinders AVTok from exploiting aligned audio-visual information, we devise a hierarchical training strategy to progressively realize reconstruction capabilities for each modality. Extensive experiments demonstrate that AVTok excels both in audio-video reconstruction and when integrated into downstream pipelines for audio-to-video, video-to-audio, and class-conditional joint audio-video generation. AVTok paves the way for the challenge of joint audio-video tokenization and provides a potential direction to build unified large multimodal models for audio-video generation.