ProMSA：用於基於知識之視覺問答的漸進式多模態搜尋代理

摘要

知識型視覺問答（KB-VQA）要求模型結合圖像理解與外部知識。多數現有方法採用固定的「檢索-生成」流程，搭配預選的檢索器與靜態的 top-k 設定，在推理過程中缺乏適應性。我們提出 ProMSA，一種用於 KB-VQA 的漸進式多模態搜尋智能體。給定一組圖像-問題配對，該智能體在明確的工具呼叫預算下，反覆選擇「圖像搜尋」、「文字搜尋」或「停止」，並透過去重機制避免冗餘檢索。在訓練方面，我們首先使用拒絕取樣監督微調（拒絕取樣 SFT）學習有效的工具使用格式，接著以 TN-GSPO（一種序列層級強化學習目標，同時根據生成長度與工具互動深度對更新進行正規化）最佳化智能體。在 E-VQA 與 InfoSeek 上的實驗顯示，該方法在強大的檢索增強生成（RAG）與智能體基準方法上取得一致提升，同時改善檢索與端到端準確率。程式碼已公開於 https://github.com/DingWu1021/Promsa。

English

Knowledge-based Visual Question Answering (KB-VQA) requires models to combine image understanding with external knowledge. Most prior methods use a fixed retrieve-then-generate pipeline with a pre-selected retriever and a static top-k setting, which is not adaptive during reasoning. We propose ProMSA, a progressive multimodal search agent for KB-VQA. Given an image-question pair, the agent iteratively chooses image search, text search, or stop, under explicit tool-call budgets and with deduplication to avoid redundant retrieval. For training, we first use rejection-sampling SFT to learn valid tool-use formats, then optimize the agent with TN-GSPO, a sequence-level RL objective that normalizes updates by both generation length and tool-interaction depth. Experiments on E-VQA and InfoSeek show consistent gains over strong RAG and agent baselines, and improved retrieval and end-to-end accuracy. The code is available at https://github.com/DingWu1021/Promsa.