2024年CONDA共享任務的數據污染報告

Data Contamination Report from the 2024 CONDA Shared Task

July 31, 2024

作者: Oscar Sainz, Iker García-Ferrero, Alon Jacovi, Jon Ander Campos, Yanai Elazar, Eneko Agirre, Yoav Goldberg, Wei-Lin Chen, Jenny Chim, Leshem Choshen, Luca D'Amico-Wong, Melissa Dell, Run-Ze Fan, Shahriar Golchin, Yucheng Li, Pengfei Liu, Bhavish Pahwa, Ameya Prabhu, Suryansh Sharma, Emily Silcock, Kateryna Solonko, David Stap, Mihai Surdeanu, Yu-Min Tseng, Vishaal Udandarao, Zengzhi Wang, Ruijie Xu, Jinglin Yang

cs.AI

摘要

第一屆資料污染研討會（CONDA 2024）專注於自然語言處理中資料污染的所有相關方面，其中資料污染被理解為評估資料包含在用於訓練大型模型的預訓練語料庫中的情況，從而影響評估結果。該研討會促進了一項共享任務，以收集有關當前可用數據集和模型中的資料污染的證據。共享任務和相關數據庫的目標是幫助社區了解問題的程度，並幫助研究人員避免在已知受污染資源上報告評估結果。共享任務提供了一個結構化的、集中的公共數據庫，用於收集污染證據，歡迎社區通過GitHub的pool requests做出貢獻。這份首次編輯論文基於來自總共23位貢獻者的91個受污染來源的566個報告條目。個別污染事件的詳細信息可在平台上找到。該平台仍然在線，歡迎社區做出貢獻。

English

The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models. The goal of the shared task and associated database is to assist the community in understanding the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. The shared task provides a structured, centralized public database for the collection of contamination evidence, open to contributions from the community via GitHub pool requests. This first compilation paper is based on 566 reported entries over 91 contaminated sources from a total of 23 contributors. The details of the individual contamination events are available in the platform. The platform continues to be online, open to contributions from the community.