Mage-VL：一種高效的編解碼器原生串流多模態基礎模型

摘要

标准视觉-语言模型（VLM）面临莫拉维克悖论：它们擅长复杂的离线视觉推理，但在简单的流式感知任务上表现不佳且处理效率低下。我们提出Mage-VL，一种高效的编解码器原生流式基础模型，专为实时多模态理解与交互设计。其核心在于自定义分词器Mage-ViT，该分词器通过利用运动向量和残差能量，在稀疏锚点帧（I帧）和预测帧（P帧）上选择性编码动态、高熵区域，取代均匀帧采样。在16×16块级别上运行时，该技术将视觉令牌消耗降低超过75%，同时保留时空上下文。Mage-ViT从头训练于约5.6亿张无标签图像和1亿个无标签视频帧，其性能匹敌或超越在数十亿图像-文本对上训练的旗舰编码器。我们建立了AI4AI数据管道，涵盖提示词-代码联合优化的多模态描述生成，以及AI驱动的性能诊断以指导训练策略。此外，通过受生物启发的双系统架构——轻量级系统1事件门控与因果系统2解码器——Mage-VL实现了主动式流式感知。大量评估表明，Mage-VL-4B在静态任务上与Qwen3-VL-4B性能相当，同时在视频理解与2D/3D空间推理上取得显著提升，推理速度提升高达3.5倍（墙钟时间），并全面超越15B参数的Phi-4-reasoning-vision基线。除模型本身外，我们贡献了七项关键实证发现，涵盖预训练数据效率、可变分辨率缩放、编解码器系统加速、VideoQA SFT冗余性、运动-空间协同、AI4AI数据管道，以及用于多模态强化学习的零视觉微调（Zero-Vision SFT）。

English

Standard vision-language models (VLMs) suffer from Moravec's paradox: they excel at complex offline visual reasoning but struggle with simple streaming perception tasks and process them inefficiently. We present Mage-VL, an efficient codec-native streaming foundation model for real-time multimodal understanding and interaction. At its core, our custom tokenizer, Mage-ViT, replaces uniform frame sampling by selectively encoding dynamic, entropy-rich regions using motion vectors and residual energy across sparse anchor (I) and predicted (P) frames. Operating at a 16 x 16 patch level, this reduces visual token consumption by over 75% while preserving spatiotemporal context. Trained from scratch on approximately 560M unlabeled images and 100M unlabeled video frames, Mage-ViT matches or outperforms flagship encoders trained on billions of image-text pairs. We establish AI4AI data pipelines encompassing prompt-code joint optimization for multimodal captioning and AI-driven performance diagnosis to guide training recipes. Furthermore, through a bio-inspired dual-system architecture - a lightweight System 1 event gate and a causal System 2 decoder - Mage-VL enables proactive streaming perception. Extensive evaluations show that Mage-VL-4B matches Qwen3-VL-4B on static tasks while achieving strong gains in video understanding and 2D/3D spatial reasoning, with up to a 3.5x wall-clock inference speedup, and comprehensively surpasses the 15B Phi-4-reasoning-vision baseline. Beyond model artifacts, we deliver seven key empirical findings covering pre-training data efficiency, variable-resolution scaling, codec system acceleration, VideoQA SFT redundancy, motion-spatial synergy, AI4AI data pipelines, and Zero-Vision SFT for multimodal RL.