PEBench：一个用于评估多模态大语言模型机器遗忘性能的虚构数据集

PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models

March 16, 2025

作者: Zhaopan Xu, Pengfei Zhou, Weidong Tang, Jiaxin Ai, Wangbo Zhao, Xiaojiang Peng, Kai Wang, Yang You, Wenqi Shao, Hongxun Yao, Kaipeng Zhang

cs.AI

摘要

近年来，多模态大语言模型（MLLMs）在视觉问答、视觉理解及推理等任务中展现出显著进步。然而，这一令人瞩目的进展依赖于从互联网收集的海量数据，引发了关于隐私与安全的重大关切。为解决这些问题，机器遗忘（MU）作为一种有前景的解决方案应运而生，它能够在无需从头训练的情况下，从已训练模型中移除特定知识。尽管MLLMs的MU已引起关注，但当前对其有效性的评估仍不全面，且基础问题常定义不清，这阻碍了开发更安全、可信赖系统的策略。为填补这一空白，我们引入了一个名为PEBench的基准，它包含个人实体及相应通用事件场景的数据集，旨在全面评估MLLMs的MU性能。通过PEBench，我们期望提供一个标准化且稳健的框架，以推动安全与隐私保护多模态模型的研究。我们对6种MU方法进行了基准测试，揭示了它们的优势与局限，并阐明了MLLMs中MU面临的关键挑战与机遇。

English

In recent years, Multimodal Large Language Models (MLLMs) have demonstrated remarkable advancements in tasks such as visual question answering, visual understanding, and reasoning. However, this impressive progress relies on vast amounts of data collected from the internet, raising significant concerns about privacy and security. To address these issues, machine unlearning (MU) has emerged as a promising solution, enabling the removal of specific knowledge from an already trained model without requiring retraining from scratch. Although MU for MLLMs has gained attention, current evaluations of its efficacy remain incomplete, and the underlying problem is often poorly defined, which hinders the development of strategies for creating more secure and trustworthy systems. To bridge this gap, we introduce a benchmark, named PEBench, which includes a dataset of personal entities and corresponding general event scenes, designed to comprehensively assess the performance of MU for MLLMs. Through PEBench, we aim to provide a standardized and robust framework to advance research in secure and privacy-preserving multimodal models. We benchmarked 6 MU methods, revealing their strengths and limitations, and shedding light on key challenges and opportunities for MU in MLLMs.