PEBench：一個虛構數據集，用於評測多模態大型語言模型的機器遺忘能力

PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models

March 16, 2025

作者: Zhaopan Xu, Pengfei Zhou, Weidong Tang, Jiaxin Ai, Wangbo Zhao, Xiaojiang Peng, Kai Wang, Yang You, Wenqi Shao, Hongxun Yao, Kaipeng Zhang

cs.AI

摘要

近年來，多模態大型語言模型（MLLMs）在視覺問答、視覺理解及推理等任務上展現了顯著的進展。然而，這一令人印象深刻的進步依賴於從網路上收集的大量數據，這引發了對隱私和安全的重要擔憂。為解決這些問題，機器遺忘（MU）作為一種有前景的解決方案應運而生，它能夠從已訓練的模型中移除特定知識，而無需從頭開始重新訓練。儘管MLLMs的MU已引起關注，但目前對其效能的評估仍不完整，且基本問題往往定義不清，這阻礙了開發更安全、更可信系統的策略。為彌補這一差距，我們引入了一個名為PEBench的基準，其中包括個人實體及相應一般事件場景的數據集，旨在全面評估MLLMs的MU性能。通過PEBench，我們希望提供一個標準化且穩健的框架，以推動安全和隱私保護的多模態模型研究。我們對6種MU方法進行了基準測試，揭示了它們的優勢與局限，並為MLLMs中的MU關鍵挑戰和機遇提供了洞見。

English

In recent years, Multimodal Large Language Models (MLLMs) have demonstrated remarkable advancements in tasks such as visual question answering, visual understanding, and reasoning. However, this impressive progress relies on vast amounts of data collected from the internet, raising significant concerns about privacy and security. To address these issues, machine unlearning (MU) has emerged as a promising solution, enabling the removal of specific knowledge from an already trained model without requiring retraining from scratch. Although MU for MLLMs has gained attention, current evaluations of its efficacy remain incomplete, and the underlying problem is often poorly defined, which hinders the development of strategies for creating more secure and trustworthy systems. To bridge this gap, we introduce a benchmark, named PEBench, which includes a dataset of personal entities and corresponding general event scenes, designed to comprehensively assess the performance of MU for MLLMs. Through PEBench, we aim to provide a standardized and robust framework to advance research in secure and privacy-preserving multimodal models. We benchmarked 6 MU methods, revealing their strengths and limitations, and shedding light on key challenges and opportunities for MU in MLLMs.