多模态结构生成：CVPR 第二届 MMFM 挑战赛技术报告

Multimodal Structured Generation: CVPR's 2nd MMFM Challenge Technical Report

June 17, 2024

作者: Franz Louis Cesista

cs.AI

摘要

多模基础模型（MMFMs）在各种计算机视觉和自然语言处理任务上展现出卓越的性能。然而，在特定任务，如文档理解方面，它们的性能仍然有限。相比传统的单模型，它们在微调和部署过程中需要更多的计算、时间和工程资源。在本报告中，我们提出了多模结构生成，这是一个通用框架，它约束冻结的MMFMs的输出logits，迫使它们在用结构化输出进行推理后再做出响应，以便下游API可以解析和使用。我们详细介绍了我们的方法，包括技术细节、理论讨论以及在由计算机视觉与模式识别（CVPR）会议主办的第二届多模基础模型挑战中的最终评估结果。我们的方法在第二阶段的隐藏测试集中取得了第二高的分数，总体排名第三。这显示了该方法能够泛化到未见过的任务。正如我们在我们的论文《检索增强结构生成：商业文档信息提取作为工具使用》中首次讨论的那样，简单的工程方法可以击败昂贵且复杂的建模步骤。我们所有的脚本、部署步骤和评估结果都可以在https://github.com/leloykun/MMFM-Challenge 上获取。

English

Multimodal Foundation Models (MMFMs) have shown remarkable performance on various computer vision and natural language processing tasks. However, their performance on particular tasks such as document understanding is still limited. They also require more compute, time, and engineering resources to finetune and deploy compared to traditional, unimodal models. In this report, we present Multimodal Structured Generation, a general framework which constrains the output logits of frozen MMFMs to force them to reason before responding with structured outputs that downstream APIs can parse and use. We provide a detailed account of our approach, including the technical details, theoretical discussions, and final evaluation results in the 2nd Multimodal Foundation Models Challenge hosted by the Computer Vision and Pattern Recognition (CVPR) conference. Our approach achieved the second highest score in the hidden test set for Phase 2 and third highest overall. This shows the method's ability to generalize to unseen tasks. And that simple engineering can beat expensive & complicated modelling steps as we first discussed in our paper, Retrieval Augmented Structured Generation: Business Document Information Extraction as Tool Use. All of our scripts, deployment steps, and evaluation results can be accessed in https://github.com/leloykun/MMFM-Challenge