LOKI：使用大型多模型模型的綜合合成數據檢測基準。

LOKI: A Comprehensive Synthetic Data Detection Benchmark using Large Multimodal Models

October 13, 2024

作者: Junyan Ye, Baichuan Zhou, Zilong Huang, Junan Zhang, Tianyi Bai, Hengrui Kang, Jun He, Honglin Lin, Zihao Wang, Tong Wu, Zhizheng Wu, Yiping Chen, Dahua Lin, Conghui He, Weijia Li

cs.AI

摘要

隨著人工智慧生成內容的快速發展，未來的互聯網可能會被合成數據淹沒，使得區分真實可信的多模態數據變得日益具有挑戰性。因此，合成數據檢測引起了廣泛關注，大型多模態模型（LMMs）在此任務中的表現引起了顯著興趣。LMMs能夠提供其真實性判斷的自然語言解釋，增強了對合成內容檢測的可解釋性。同時，區分真實和合成數據的任務有效地測試了LMMs的感知、知識和推理能力。為此，我們引入了LOKI，一個旨在評估LMMs跨多模態檢測合成數據能力的新型基準。LOKI包括視頻、圖像、3D、文本和音頻模態，涵蓋了26個子類別的18,000個精心策劃問題，具有明確的難度級別。該基準包括粗粒度判斷和多選問題，以及細粒度異常選擇和解釋任務，可進行對LMMs的全面分析。我們在LOKI上評估了22個開源LMMs和6個封閉源模型，突出它們作為合成數據檢測器的潛力，同時也揭示了LMM能力發展中的一些限制。有關LOKI的更多信息，請訪問https://opendatalab.github.io/LOKI/。

English

With the rapid development of AI-generated content, the future internet may be inundated with synthetic data, making the discrimination of authentic and credible multimodal data increasingly challenging. Synthetic data detection has thus garnered widespread attention, and the performance of large multimodal models (LMMs) in this task has attracted significant interest. LMMs can provide natural language explanations for their authenticity judgments, enhancing the explainability of synthetic content detection. Simultaneously, the task of distinguishing between real and synthetic data effectively tests the perception, knowledge, and reasoning capabilities of LMMs. In response, we introduce LOKI, a novel benchmark designed to evaluate the ability of LMMs to detect synthetic data across multiple modalities. LOKI encompasses video, image, 3D, text, and audio modalities, comprising 18K carefully curated questions across 26 subcategories with clear difficulty levels. The benchmark includes coarse-grained judgment and multiple-choice questions, as well as fine-grained anomaly selection and explanation tasks, allowing for a comprehensive analysis of LMMs. We evaluated 22 open-source LMMs and 6 closed-source models on LOKI, highlighting their potential as synthetic data detectors and also revealing some limitations in the development of LMM capabilities. More information about LOKI can be found at https://opendatalab.github.io/LOKI/