通過程式碼生成的模組化視覺問答

Modular Visual Question Answering via Code Generation

June 8, 2023

作者: Sanjay Subramanian, Medhini Narasimhan, Kushal Khangaonkar, Kevin Yang, Arsha Nagrani, Cordelia Schmid, Andy Zeng, Trevor Darrell, Dan Klein

cs.AI

摘要

我們提出了一個將視覺問答形式化為模塊化代碼生成的框架。與先前關於VQA模塊化方法的工作相比，我們的方法無需額外訓練，依賴於預訓練的語言模型（LMs）、在圖像說明對上預訓練的視覺模型，以及用於上下文學習的五十個VQA範例。生成的Python程序使用算術和條件邏輯調用和組合視覺模型的輸出。相較於未使用代碼生成的少樣本基線，我們的方法在COVR數據集上將準確性提高至少3％，在GQA數據集上提高約2％。

English

We present a framework that formulates visual question answering as modular code generation. In contrast to prior work on modular approaches to VQA, our approach requires no additional training and relies on pre-trained language models (LMs), visual models pre-trained on image-caption pairs, and fifty VQA examples used for in-context learning. The generated Python programs invoke and compose the outputs of the visual models using arithmetic and conditional logic. Our approach improves accuracy on the COVR dataset by at least 3% and on the GQA dataset by roughly 2% compared to the few-shot baseline that does not employ code generation.