知道何事不該回答：視覺-語言模型中的選擇性不遵從

摘要

預期視覺語言模型（VLM）應對恰當的請求提供有幫助的回應，同時對不正確、不安全、不可行或無法回答的請求不予遵從。然而，現有基準大多在整個查詢層級評估不遵從行為，假設每個請求要不是應予遵從，就是應予以拒絕遵從。實際上，真實世界的查詢可能夾雜可回答的內容與應拒絕遵從的組成部分。在本文中，我們介紹 KoNA，這是一個用於評估 VLM 在五個類別中選擇性不遵從行為的基準：錯誤前提、視覺不可及性、普遍未知、任務可行性與安全性。每個任務在成對的單一與複合查詢下，評估兩種能力：查詢層級的不遵從與成分層級的不遵從。我們對多種 VLM 的評估顯示，模型經常無法適當地拒絕、糾正或棄答，而當查詢需要選擇性不遵從時，這些失敗更加明顯。為了解決這項挑戰，我們使用需要選擇性不遵從的 KoNA 範例，連同一組應直接給出答案的完全可回答範例，對 VLM 進行微調。微調後的模型在不遵從準確率上獲得大幅提升，同時在完全可回答的任務上大致維持效能。這些結果表明，微調後的模型能區分可回答的組成部分與需要不遵從的組成部分，並以符合任務的方式回應。

English

Vision-language models (VLMs) are expected to respond helpfully to appropriate requests while withholding compliance with requests that are incorrect, unsafe, infeasible, or unanswerable. However, existing benchmarks predominantly evaluate non-compliance at the level of the query as a whole, assuming that each request either warrants compliance or requires withholding compliance. In practice, real-world queries can contain a mixture of answerable content and components for which compliance should be withheld. In this paper, we introduce KoNA, a benchmark for evaluating selective non-compliance in VLMs across five categories: False Premise, Visual Inaccessibility, Universal Unknown, Task Feasibility, and Safety. Each task evaluates two capabilities: query-level non-compliance and component-level non-compliance under paired single and compound queries. Our evaluation across diverse VLMs shows that models often fail to refuse, correct, or abstain appropriately, and these failures become more pronounced when queries require selective non-compliance. To address this challenge, we fine-tune VLMs using KoNA examples that require selective non-compliance, together with a fully answerable set that should receive direct answers. Our fine-tuned models achieve substantial improvements in non-compliance accuracy while largely maintaining performance on fully answerable tasks. These results suggest that the fine-tuned models can distinguish between answerable components and those requiring non-compliance and respond in a task-appropriate manner.