MonkeyOCRv2：用於文檔AI的視覺-文字基礎模型

摘要

主流視覺編碼器預訓練於自然圖像，未經面向文檔的適配便無法有效應用於文檔圖像，因為密集文字與細粒度的字符筆畫需要字符層級的視覺感知。我們提出 MonkeyOCRv2，這是一個專為文檔人工智慧設計的視覺-文字預訓練模型。首先，我們構建了 MonkeyDoc v2，據我們所知，這是目前最大的文檔圖像預訓練語料庫，包含 1.13 億張圖像，涵蓋 17 種語言。其次，我們提出了一種聯合學習圖像到文字生成與像素級文檔重建的預訓練策略：前者使視覺表徵與文字內容對齊，後者則保留字符筆畫與佈局細節。我們在五項具代表性的文檔分析任務上進行了大量實驗，包括文字識別、公式識別、文字檢測、文檔篡改檢測以及重疊文字分割。將原始編碼器替換為 MonkeyOCRv2 後，所有五項任務的性能均獲得持續提升。最後，我們在更具挑戰性的文檔解析與文檔理解任務中，驗證了其作為多模態大語言模型視覺編碼器的有效性。當保持凍結狀態並與輕量級語言模型配對時，它產生了僅 0.7B 參數的文檔解析模型，在 MDPBench（一個近期推出的基準測試，涵蓋數位原生與拍攝的文檔，跨越 17 種語言）上創下了新的開源最佳表現，以 2.8% 的絕對值超越了先前最佳的 3B 參數 dots.mocr 模型，而視覺編碼器大小約為其 1/11。該凍結編碼器亦驅動了一個文檔理解模型，在相同訓練設定下，於八個基準測試中超越了基於 CLIP、DINO 和 SAM 的同類模型。這些結果表明，面向文檔的視覺預訓練本身就足以成為文檔智能的基礎。

English

Mainstream visual encoders are pretrained on natural images and cannot be effectively applied to document images without document-oriented adaptation, as dense text and fine-grained character strokes demand character-level visual perception. We present MonkeyOCRv2, a visual-text pretrained model for document AI. First, we construct MonkeyDoc v2, to our knowledge the largest document-image pretraining corpus, comprising 113 million images spanning 17 languages. Second, we propose a pretraining strategy that jointly learns image-to-text generation and pixel-level document reconstruction: the former aligns visual representations with textual content, while the latter preserves character strokes and layout details. Extensive experiments are conducted on five representative document analysis tasks, including text recognition, formula recognition, text detection, document tampering detection, and overlapping text segmentation. Replacing the original encoders with MonkeyOCRv2 consistently improves performance across all five tasks. Finally, we validate its effectiveness as the vision encoder of multimodal large language models on the more challenging tasks of document parsing and document understanding. Kept frozen and paired with a lightweight language model, it yields a 0.7B document parsing model that sets a new open-source state-of-the-art on MDPBench, a recent benchmark spanning digital-born and photographed documents across 17 languages, surpassing the previous best 3B dots.mocr by 2.8% absolute with a vision encoder roughly 11times smaller. The frozen encoder also powers a document understanding model that outperforms counterparts built on CLIP, DINO, and SAM across eight benchmarks under identical training settings. These results suggest that document-oriented visual pretraining can serve as a foundation for document intelligence in its own right.