视觉说服力：什么在影响视觉语言模型的决策？

Visual Persuasion: What Influences Decisions of Vision-Language Models?

February 17, 2026

作者: Manuel Cherep, Pranav M R, Pattie Maes, Nikhil Singh

cs.AI

摘要

网络上充斥着大量原本为人类消费而创建的图像，如今这些图像正日益被基于视觉语言模型（VLM）的智能体所解析。这些智能体大规模地做出视觉决策，决定点击、推荐或购买哪些内容。然而，我们对其视觉偏好结构知之甚少。我们提出一个研究框架：将VLM置于受控的基于图像的选择任务中，并系统性地扰动其输入。其核心思想是将智能体的决策函数视为潜在视觉效用，可通过显示性偏好（即对经过系统性编辑的图像进行选择）来推断。从商品图片等常见图像出发，我们提出视觉提示优化方法，通过适配文本优化技术，利用图像生成模型迭代式地提出并应用视觉合理的修改（如构图、光线或背景）。随后评估哪些编辑能提升被选概率。通过对前沿VLM的大规模实验，我们证明优化后的编辑能在直接比较中显著改变选择概率。我们还开发了自动可解释性管道来解释这些偏好，识别驱动选择行为的一致视觉主题。我们认为该方法能有效揭示视觉漏洞和安全隐患——这些隐患若在自然场景中被动发现可能造成更大风险，从而为基于图像的AI智能体提供更主动的审计与治理方案。

English

The web is littered with images, once created for human consumption and now increasingly interpreted by agents using vision-language models (VLMs). These agents make visual decisions at scale, deciding what to click, recommend, or buy. Yet, we know little about the structure of their visual preferences. We introduce a framework for studying this by placing VLMs in controlled image-based choice tasks and systematically perturbing their inputs. Our key idea is to treat the agent's decision function as a latent visual utility that can be inferred through revealed preference: choices between systematically edited images. Starting from common images, such as product photos, we propose methods for visual prompt optimization, adapting text optimization methods to iteratively propose and apply visually plausible modifications using an image generation model (such as in composition, lighting, or background). We then evaluate which edits increase selection probability. Through large-scale experiments on frontier VLMs, we demonstrate that optimized edits significantly shift choice probabilities in head-to-head comparisons. We develop an automatic interpretability pipeline to explain these preferences, identifying consistent visual themes that drive selection. We argue that this approach offers a practical and efficient way to surface visual vulnerabilities, safety concerns that might otherwise be discovered implicitly in the wild, supporting more proactive auditing and governance of image-based AI agents.