SeeGULL：一个具有广泛地理文化覆盖范围的刻板印象基准利用生成模型

摘要

在 NLP 模型中，刻板印象基准数据集对于检测和减轻关于人群的社会刻板印象至关重要。然而，现有数据集在规模和覆盖范围上存在限制，并且主要局限于西方社会中普遍存在的刻板印象。随着语言技术在全球范围内的普及，这一问题尤为严重。为了填补这一空白，我们提出了SeeGULL，一个广覆盖的刻板印象数据集，利用诸如PaLM和GPT-3等大型语言模型的生成能力构建，并利用全球多样化的评分人群验证这些刻板印象在社会中的普遍程度。SeeGULL 以英语为主，包含涵盖178个国家、8个不同地缘政治地区、6大洲的身份群体的刻板印象，以及美国和印度境内的州级身份认同。我们还为不同刻板印象包括细粒度的冒犯程度评分，并展示它们之间的全球差异。此外，我们还包括了对同一群体的比较注释，其中注释者分别居住在该地区和北美，展示了地区内关于群体的刻板印象与北美普遍存在的刻板印象之间的差异。内容警告：本文包含可能具有冒犯性的刻板印象示例。

English

Stereotype benchmark datasets are crucial to detect and mitigate social stereotypes about groups of people in NLP models. However, existing datasets are limited in size and coverage, and are largely restricted to stereotypes prevalent in the Western society. This is especially problematic as language technologies gain hold across the globe. To address this gap, we present SeeGULL, a broad-coverage stereotype dataset, built by utilizing generative capabilities of large language models such as PaLM, and GPT-3, and leveraging a globally diverse rater pool to validate the prevalence of those stereotypes in society. SeeGULL is in English, and contains stereotypes about identity groups spanning 178 countries across 8 different geo-political regions across 6 continents, as well as state-level identities within the US and India. We also include fine-grained offensiveness scores for different stereotypes and demonstrate their global disparities. Furthermore, we include comparative annotations about the same groups by annotators living in the region vs. those that are based in North America, and demonstrate that within-region stereotypes about groups differ from those prevalent in North America. CONTENT WARNING: This paper contains stereotype examples that may be offensive.

SeeGULL：一个具有广泛地理文化覆盖范围的刻板印象基准利用生成模型

SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models

摘要

Support