VLMGuard：通过未标记数据防御VLMs免受恶意提示

摘要

视觉语言模型（VLMs）对于上下文理解视觉和文本信息至关重要。然而，它们对恶意操纵输入的脆弱性存在重大风险，导致输出受损，并引发对VLM集成应用可靠性的担忧。因此，检测这些恶意提示对于维护对VLM生成的信任至关重要。在开发安全提示分类器时面临的主要挑战是缺乏大量标记的良性和恶意数据。为解决这一问题，我们引入了VLMGuard，这是一种新颖的学习框架，利用野外未标记的用户提示进行恶意提示检测。这些未标记的提示在VLM在开放世界中部署时自然产生，包含良性和恶意信息。为了利用这些未标记数据，我们提出了一种自动恶意估计分数，用于区分这些未标记混合中的良性和恶意样本，从而实现在其上训练二元提示分类器。值得注意的是，我们的框架不需要额外的人工注释，为实际应用提供了强大的灵活性和实用性。广泛的实验表明，VLMGuard实现了卓越的检测结果，明显优于最先进的方法。免责声明：本文可能包含冒犯性示例；请谨慎阅读。

English

Vision-language models (VLMs) are essential for contextual understanding of both visual and textual information. However, their vulnerability to adversarially manipulated inputs presents significant risks, leading to compromised outputs and raising concerns about the reliability in VLM-integrated applications. Detecting these malicious prompts is thus crucial for maintaining trust in VLM generations. A major challenge in developing a safeguarding prompt classifier is the lack of a large amount of labeled benign and malicious data. To address the issue, we introduce VLMGuard, a novel learning framework that leverages the unlabeled user prompts in the wild for malicious prompt detection. These unlabeled prompts, which naturally arise when VLMs are deployed in the open world, consist of both benign and malicious information. To harness the unlabeled data, we present an automated maliciousness estimation score for distinguishing between benign and malicious samples within this unlabeled mixture, thereby enabling the training of a binary prompt classifier on top. Notably, our framework does not require extra human annotations, offering strong flexibility and practicality for real-world applications. Extensive experiment shows VLMGuard achieves superior detection results, significantly outperforming state-of-the-art methods. Disclaimer: This paper may contain offensive examples; reader discretion is advised.

VLMGuard：通过未标记数据防御VLMs免受恶意提示

VLMGuard: Defending VLMs against Malicious Prompts via Unlabeled Data

摘要

Summary

Support

Support