KVAE：面向多模态生成模型的系列分词器

摘要

潜在扩散建模（LDM）作为一种重要范式，利用分词器将输入信号映射为压缩表示。这种依赖关系使分词器成为生成过程本身不可或缺的一部分，因为它影响学习速度、合成样本的质量，并为后续应用奠定基础。本报告介绍了一系列用于音频、图像和视频的KVAE分词器，所有模型均设计用于后续的文本条件生成：KVAE-Audio，一种连续全频带48 kHz分词器，具有50 Hz的64通道潜表示；KVAE-3D——两种因果视频分词器，分别实现4x16x16和4x8x8压缩；KVAE-2D，一种图像模型，以32通道将输入压缩8倍。我们证明，在重建指标（PSNR、LPIPS、PESQ等）以及生成结果的客观指标（弗雷歇距离、CLIP分数、CLAP分数等）和主观指标（并排对比评估）上，本方法匹配或超越了前沿开源分词器，例如来自Wan-2.2、HunyuanVideo-1.5、FLUX.2、MovieGen、StableAudio和MMAudio的VAE。考虑到开发难度，我们与社区分享训练细节、模型选择方法以及设计选择上的消融实验。代码公开于https://github.com/kandinskylab/kvae 和 https://github.com/kandinskylab/kvae-audio。

English

Latent diffusion modeling (LDM), a prominent paradigm, utilizes tokenizers to map input signal to compressed representation. This dependency positions tokenizer as an integral part of generation process itself, since it affects learning speed, quality of synthesized samples and lay foundation for later applications. This report presents series of KVAE tokenizers for audio, image and video, all designed for subsequent text-conditioned generation: KVAE-Audio, a continuous full-band 48 kHz tokenizer with a 50 Hz latent of 64 channels; KVAE-3D -- two causal video tokenizers for 4x16x16 and 4x8x8 compression; KVAE-2D, an image model, compressing input by factor of 8 with 32 channels. We demonstrate that reconstruction (PSNR, LPIPS, PESQ, etc.) and generation results on objective (Frechet Distance, CLIP score, CLAP score, etc.) and subjective (side-by-side evaluation) metrics matches or surpasses frontier opensource tokenizers, such as VAEs from Wan-2.2, HunyuanVideo-1.5, FLUX.2, MovieGen, StableAudio and MMAudio. Considering difficulty of development, we share with community training details, model selection method and ablation on design choices. The code is publicly available at https://github.com/kandinskylab/kvae and https://github.com/kandinskylab/kvae-audio.