JavisGPT：面向音视频理解与生成的统一多模态大语言模型

摘要

本文提出JavisGPT——首个面向音视频联合理解与生成任务的多模态大语言模型。该模型采用简洁的编码器-LLM-解码器架构，通过同步融合模块实现时空音视频特征融合，并利用同步感知可学习查询桥接预训练的音视频扩散Transformer生成器，从而基于多模态指令实现时序一致的多模态理解与生成。我们设计了三阶段训练流程：多模态预训练、音视频微调与大规模指令调优，逐步增强现有视觉语言模型的多模态能力。为支撑训练，我们构建了包含20万条GPT-4o标注音视频文本对话的高质量指令集JavisInst-Omni，涵盖多样化、多层级的理解与生成场景。在音视频理解与生成基准测试中，JavisGPT显著优于现有多模态大模型，尤其在复杂时序同步任务中表现突出。

English

This paper presents JavisGPT, the first unified multimodal large language model (MLLM) for Joint Audio-Video (JAV) comprehension and generation. JavisGPT adopts a concise encoder-LLM-decoder architecture, featuring a SyncFusion module for spatio-temporal audio-video fusion and synchrony-aware learnable queries to bridge a pretrained JAV-DiT generator. This design enables temporally coherent video-audio understanding and generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuning, to progressively build multimodal comprehension and generation from existing vision-language models. To support this, we further construct JavisInst-Omni, a high-quality instruction dataset with over 200K GPT-4o-curated audio-video-text dialogues that span diverse and multi-level comprehension and generation scenarios. Extensive experiments on JAV comprehension and generation benchmarks show that JavisGPT outperforms existing MLLMs, particularly in complex and temporally synchronized settings.

JavisGPT：面向音视频理解与生成的统一多模态大语言模型

JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation

摘要

Support