多模態結構生成：CVPR 第二屆 MMFM 挑戰賽技術報告

Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical Report

June 17, 2024

作者: Franz Louis Cesista

cs.AI

摘要

多模基礎模型（MMFMs）在各種計算機視覺和自然語言處理任務中展現出卓越的表現。然而，它們在特定任務上，如文件理解方面的表現仍然有限。相較於傳統的單模型，它們還需要更多的計算、時間和工程資源來進行微調和部署。在本報告中，我們提出了多模結構生成，這是一個通用框架，將凍結的MMFMs的輸出logits限制在強迫它們在回應之前進行推理，以生成結構化輸出，下游API可以解析和使用。我們詳細介紹了我們的方法，包括技術細節、理論討論以及在由計算機視覺和模式識別（CVPR）會議主辦的第2屆多模基礎模型挑戰中的最終評估結果。我們的方法在第2階段的隱藏測試集中取得了第二高的分數，整體排名第三。這顯示了該方法對未見任務的泛化能力。正如我們在我們的論文《檢索增強結構生成：商業文件信息提取作為工具使用》中首次討論的那樣，簡單的工程方法可以擊敗昂貴和複雜的建模步驟。我們所有的腳本、部署步驟和評估結果都可以在https://github.com/leloykun/MMFM-Challenge中找到。

English

Multimodal Foundation Models (MMFMs) have shown remarkable performance on various computer vision and natural language processing tasks. However, their performance on particular tasks such as document understanding is still limited. They also require more compute, time, and engineering resources to finetune and deploy compared to traditional, unimodal models. In this report, we present Multimodal Structured Generation, a general framework which constrains the output logits of frozen MMFMs to force them to reason before responding with structured outputs that downstream APIs can parse and use. We provide a detailed account of our approach, including the technical details, theoretical discussions, and final evaluation results in the 2nd Multimodal Foundation Models Challenge hosted by the Computer Vision and Pattern Recognition (CVPR) conference. Our approach achieved the second highest score in the hidden test set for Phase 2 and third highest overall. This shows the method's ability to generalize to unseen tasks. And that simple engineering can beat expensive & complicated modelling steps as we first discussed in our paper, Retrieval Augmented Structured Generation: Business Document Information Extraction as Tool Use. All of our scripts, deployment steps, and evaluation results can be accessed in https://github.com/leloykun/MMFM-Challenge