视频模型的视觉提示工程

摘要

在基础模型时代，模型的性能取决于其提示的质量。因此，提示工程已成为提升语言模型性能的关键技术。鉴于视频模型正逐步成为视觉任务（如视觉推理）的基础模型，我们在此探究它们是否能同样受益于视觉提示工程：即自动调整任务图像以改进模型表现。例如在视觉物理推理任务（"球经过障碍物后落在何处？"）中，可通过简单调用图像编辑模型将抽象草图场景转换为逼真版本。研究发现，视觉提示工程（简称VIPE）能显著提升各类视频推理任务的性能。事实上，对于视频模型而言，视觉提示工程的效果甚至优于传统文本提示工程或测试时扩展策略。最终，正如文本提示工程能系统性提升语言模型性能一样，视觉提示工程可作为简洁且计算高效的方案，有效激发视频模型的视觉推理能力。相关示例视频详见项目页面：https://visual-prompt-engineering.github.io/。

English

In the age of foundation models, a model is only as good as its prompt. For this reason, prompt engineering has become an essential technique for improving language model performance. Since video models are currently becoming foundation models for visual tasks (e.g., visual reasoning), we here ask whether they similarly benefit from visual prompt engineering: automatically modifying the task image to improve model performance. For example, for a visual physics reasoning task ("Where does the ball land, after passing a set of obstacles?"), an abstract sketch-like scene can be turned into a photorealistic version with a simple call to an image editing model. We find that visual prompt engineering, or VIPE for short, improves video reasoning performance across tasks. In fact, for video models, visual prompt engineering can be even more effective than classic text-based prompt engineering or test-time scaling. Ultimately, just as text-based prompt engineering systematically improves language model performance, visual prompt engineering can serve as a simple, compute-efficient approach to elicit better visual reasoning performance from video models. Example videos on our project page at https://visual-prompt-engineering.github.io/.