ProMSA：面向基于知识的视觉问答的渐进式多模态搜索代理

摘要

基于知识的视觉问答（KB-VQA）要求模型将图像理解与外部知识相结合。以往大多数方法采用固定的检索-生成流水线，使用预选检索器和静态top-k设置，在推理过程中缺乏自适应性。我们提出ProMSA，一种用于KB-VQA的渐进式多模态搜索代理。给定图像-问题对，该代理在显式工具调用预算下迭代选择图像搜索、文本搜索或停止操作，并通过去重避免冗余检索。训练方面，我们首先使用拒绝采样SFT学习有效的工具使用格式，然后通过TN-GSPO（一种序列级强化学习目标）优化代理，该目标根据生成长度和工具交互深度对更新进行归一化。在E-VQA和InfoSeek上的实验表明，该方法在强RAG和代理基线上取得了持续改进，并提升了检索和端到端准确率。代码开源地址：https://github.com/DingWu1021/Promsa。

English

Knowledge-based Visual Question Answering (KB-VQA) requires models to combine image understanding with external knowledge. Most prior methods use a fixed retrieve-then-generate pipeline with a pre-selected retriever and a static top-k setting, which is not adaptive during reasoning. We propose ProMSA, a progressive multimodal search agent for KB-VQA. Given an image-question pair, the agent iteratively chooses image search, text search, or stop, under explicit tool-call budgets and with deduplication to avoid redundant retrieval. For training, we first use rejection-sampling SFT to learn valid tool-use formats, then optimize the agent with TN-GSPO, a sequence-level RL objective that normalizes updates by both generation length and tool-interaction depth. Experiments on E-VQA and InfoSeek show consistent gains over strong RAG and agent baselines, and improved retrieval and end-to-end accuracy. The code is available at https://github.com/DingWu1021/Promsa.