SeeGULL：具有廣泛地理文化覆蓋範圍的刻板印象基準測試利用生成模型

摘要

在自然語言處理模型中，刻板印象基準數據集對於檢測和減輕有關人群的社會刻板印象至關重要。然而，現有的數據集在規模和覆蓋範圍上存在限制，並且主要僅限於西方社會中普遍存在的刻板印象。隨著語言技術在全球範圍內的應用擴大，這一問題尤為嚴重。為彌補這一差距，我們提出了SeeGULL，這是一個廣泛覆蓋的刻板印象數據集，通過利用PaLM和GPT-3等大型語言模型的生成能力構建，並利用全球多元化的評定者群體驗證這些刻板印象在社會中的普遍程度。SeeGULL以英語為主，包含關於涵蓋178個國家、8個不同地緣政治區域、6大洲的身份群體的刻板印象，以及美國和印度的州級身份。我們還為不同刻板印象包含了細緻的冒犯程度分數，展示了它們的全球差異。此外，我們還包括了對同一群體的比較標註，其中標註者分別居住在該地區和北美，並展示了地區內有關群體的刻板印象與北美普遍存在的刻板印象不同。內容警告：本文包含可能具冒犯性的刻板印象示例。

English

Stereotype benchmark datasets are crucial to detect and mitigate social stereotypes about groups of people in NLP models. However, existing datasets are limited in size and coverage, and are largely restricted to stereotypes prevalent in the Western society. This is especially problematic as language technologies gain hold across the globe. To address this gap, we present SeeGULL, a broad-coverage stereotype dataset, built by utilizing generative capabilities of large language models such as PaLM, and GPT-3, and leveraging a globally diverse rater pool to validate the prevalence of those stereotypes in society. SeeGULL is in English, and contains stereotypes about identity groups spanning 178 countries across 8 different geo-political regions across 6 continents, as well as state-level identities within the US and India. We also include fine-grained offensiveness scores for different stereotypes and demonstrate their global disparities. Furthermore, we include comparative annotations about the same groups by annotators living in the region vs. those that are based in North America, and demonstrate that within-region stereotypes about groups differ from those prevalent in North America. CONTENT WARNING: This paper contains stereotype examples that may be offensive.

SeeGULL：具有廣泛地理文化覆蓋範圍的刻板印象基準測試利用生成模型

SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models

摘要

Support