视频模型的視覺提示工程

摘要

在基础模型时代，模型性能与其提示质量成正比。因此，提示工程已成为提升语言模型性能的关键技术。鉴于视频模型正逐步成为视觉基础模型（如视觉推理），我们在此探究视觉提示工程是否能为其带来类似增益：通过自动优化任务图像来提升模型性能。例如，在视觉物理推理任务（"球体穿过障碍物后将落在何处？"）中，抽象草图场景可通过简单调用图像编辑模型转换为逼真版本。研究发现，视觉提示工程（简称VIPE）能全面提升视频推理任务的表现。事实上，对于视频模型而言，视觉提示工程甚至比传统文本提示工程或测试时缩放更为有效。最终，正如文本提示工程系统性地提升语言模型性能那般，视觉提示工程可作为一种简洁高效的计算方案，有效激发视频模型更优的视觉推理能力。项目页面示例视频请访问：https://visual-prompt-engineering.github.io/。

English

In the age of foundation models, a model is only as good as its prompt. For this reason, prompt engineering has become an essential technique for improving language model performance. Since video models are currently becoming foundation models for visual tasks (e.g., visual reasoning), we here ask whether they similarly benefit from visual prompt engineering: automatically modifying the task image to improve model performance. For example, for a visual physics reasoning task ("Where does the ball land, after passing a set of obstacles?"), an abstract sketch-like scene can be turned into a photorealistic version with a simple call to an image editing model. We find that visual prompt engineering, or VIPE for short, improves video reasoning performance across tasks. In fact, for video models, visual prompt engineering can be even more effective than classic text-based prompt engineering or test-time scaling. Ultimately, just as text-based prompt engineering systematically improves language model performance, visual prompt engineering can serve as a simple, compute-efficient approach to elicit better visual reasoning performance from video models. Example videos on our project page at https://visual-prompt-engineering.github.io/.