MonkeyOCRv2：面向文档人工智能的视觉-文本基础模型

摘要

主流视觉编码器均基于自然图像进行预训练，无法直接有效应用于文档图像，因为密集文本和精细字符笔画要求模型具备字符级视觉感知能力。本文提出MonkeyOCRv2——面向文档AI的视觉-文本预训练模型。首先，我们构建MonkeyDoc v2，据我们所知是目前最大的文档图像预训练语料库，包含1.13亿张图像，涵盖17种语言。其次，我们提出一种联合学习图像到文本生成与像素级文档重建的预训练策略：前者将视觉表征与文本内容对齐，后者保留字符笔画和布局细节。我们在五项代表性文档分析任务上开展大量实验，包括文本识别、公式识别、文本检测、文档篡改检测和重叠文本分割。将原始编码器替换为MonkeyOCRv2后，五项任务性能均持续提升。最终，我们验证其作为多模态大语言模型视觉编码器在更具挑战性的文档解析与文档理解任务中的有效性。固定该编码器并配合轻量级语言模型，即可得到参数量为0.7B的文档解析模型，其在MDPBench（近期发布的涵盖17种语言数字原生及拍摄文档的基准测试）上创下新的开源最优性能，在视觉编码器规模约为1/11的情况下，比此前最佳3B参数模型dots.mocr绝对提升2.8%。该冻结编码器还赋能一款文档理解模型，在相同训练设置下，其在八个基准测试中的表现均优于基于CLIP、DINO和SAM的同类模型。这些结果表明，面向文档的视觉预训练可独立作为文档智能的基础。

English

Mainstream visual encoders are pretrained on natural images and cannot be effectively applied to document images without document-oriented adaptation, as dense text and fine-grained character strokes demand character-level visual perception. We present MonkeyOCRv2, a visual-text pretrained model for document AI. First, we construct MonkeyDoc v2, to our knowledge the largest document-image pretraining corpus, comprising 113 million images spanning 17 languages. Second, we propose a pretraining strategy that jointly learns image-to-text generation and pixel-level document reconstruction: the former aligns visual representations with textual content, while the latter preserves character strokes and layout details. Extensive experiments are conducted on five representative document analysis tasks, including text recognition, formula recognition, text detection, document tampering detection, and overlapping text segmentation. Replacing the original encoders with MonkeyOCRv2 consistently improves performance across all five tasks. Finally, we validate its effectiveness as the vision encoder of multimodal large language models on the more challenging tasks of document parsing and document understanding. Kept frozen and paired with a lightweight language model, it yields a 0.7B document parsing model that sets a new open-source state-of-the-art on MDPBench, a recent benchmark spanning digital-born and photographed documents across 17 languages, surpassing the previous best 3B dots.mocr by 2.8% absolute with a vision encoder roughly 11times smaller. The frozen encoder also powers a document understanding model that outperforms counterparts built on CLIP, DINO, and SAM across eight benchmarks under identical training settings. These results suggest that document-oriented visual pretraining can serve as a foundation for document intelligence in its own right.