2024年CONDA共享任务的数据污染报告

Data Contamination Report from the 2024 CONDA Shared Task

July 31, 2024

作者: Oscar Sainz, Iker García-Ferrero, Alon Jacovi, Jon Ander Campos, Yanai Elazar, Eneko Agirre, Yoav Goldberg, Wei-Lin Chen, Jenny Chim, Leshem Choshen, Luca D'Amico-Wong, Melissa Dell, Run-Ze Fan, Shahriar Golchin, Yucheng Li, Pengfei Liu, Bhavish Pahwa, Ameya Prabhu, Suryansh Sharma, Emily Silcock, Kateryna Solonko, David Stap, Mihai Surdeanu, Yu-Min Tseng, Vishaal Udandarao, Zengzhi Wang, Ruijie Xu, Jinglin Yang

cs.AI

摘要

第一届数据污染研讨会（CONDA 2024）侧重于自然语言处理中数据污染的所有相关方面，其中数据污染被理解为评估数据包含在用于训练大规模模型的预训练语料库中的情况，从而损害评估结果。该研讨会促进了一个共享任务，以收集关于当前可用数据集和模型中数据污染的证据。共享任务及相关数据库的目标是帮助社区了解问题的程度，并帮助研究人员避免在已知受污染资源上报告评估结果。共享任务提供了一个结构化、集中的公共数据库，用于收集污染证据，欢迎社区通过GitHub池请求进行贡献。这篇首次汇编论文基于来自23位贡献者的总共91个受污染来源的566个报告条目。各个污染事件的详细信息可在平台上找到。该平台仍然在线，欢迎社区的贡献。

English

The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in current available datasets and models. The goal of the shared task and associated database is to assist the community in understanding the extent of the problem and to assist researchers in avoiding reporting evaluation results on known contaminated resources. The shared task provides a structured, centralized public database for the collection of contamination evidence, open to contributions from the community via GitHub pool requests. This first compilation paper is based on 566 reported entries over 91 contaminated sources from a total of 23 contributors. The details of the individual contamination events are available in the platform. The platform continues to be online, open to contributions from the community.