DIWALI - 印度文化特定项目的多样性与包容性意识：数据集及大语言模型在印度语境下文化文本适应能力的评估

DIWALI - Diversity and Inclusivity aWare cuLture specific Items for India: Dataset and Assessment of LLMs for Cultural Text Adaptation in Indian Context

September 22, 2025

作者: Pramit Sahoo, Maharaj Brahma, Maunendra Sankar Desarkar

cs.AI

摘要

大型语言模型（LLMs）在多种任务和应用中得到了广泛使用。然而，尽管其功能强大，研究表明它们因缺乏文化知识和能力而存在文化对齐不足的问题，并产生带有偏见的生成内容。由于缺乏适当的评估指标以及代表区域和次区域层面文化复杂性的文化基础数据集，评估LLMs的文化意识和对齐性尤为困难。现有的文化特定项目（CSIs）数据集主要关注区域层面的概念，且可能包含误报。为解决这一问题，我们引入了一个新颖的印度文化CSIs数据集，涵盖17个文化方面。该数据集包含来自36个次区域的sim8k文化概念。为了衡量LLMs在文化文本适应任务中的文化能力，我们利用创建的CSIs、LLM作为评判者以及来自不同社会人口区域的用户评价来评估适应效果。此外，我们进行了定量分析，展示了所有考虑的LLMs在选择性次区域覆盖和表面层次适应方面的表现。我们的数据集可在此获取：https://huggingface.co/datasets/nlip/DIWALI，项目网页链接为\href{https://nlip-lab.github.io/nlip/publications/diwali/}，我们的代码库及模型输出可在此找到：https://github.com/pramitsahoo/culture-evaluation。

English

Large language models (LLMs) are widely used in various tasks and applications. However, despite their wide capabilities, they are shown to lack cultural alignment ryan-etal-2024-unintended, alkhamissi-etal-2024-investigating and produce biased generations naous-etal-2024-beer due to a lack of cultural knowledge and competence. Evaluation of LLMs for cultural awareness and alignment is particularly challenging due to the lack of proper evaluation metrics and unavailability of culturally grounded datasets representing the vast complexity of cultures at the regional and sub-regional levels. Existing datasets for culture specific items (CSIs) focus primarily on concepts at the regional level and may contain false positives. To address this issue, we introduce a novel CSI dataset for Indian culture, belonging to 17 cultural facets. The dataset comprises sim8k cultural concepts from 36 sub-regions. To measure the cultural competence of LLMs on a cultural text adaptation task, we evaluate the adaptations using the CSIs created, LLM as Judge, and human evaluations from diverse socio-demographic region. Furthermore, we perform quantitative analysis demonstrating selective sub-regional coverage and surface-level adaptations across all considered LLMs. Our dataset is available here: https://huggingface.co/datasets/nlip/DIWALI{https://huggingface.co/datasets/nlip/DIWALI}, project webpage\href{https://nlip-lab.github.io/nlip/publications/diwali/{https://nlip-lab.github.io/nlip/publications/diwali/}}, and our codebase with model outputs can be found here: https://github.com/pramitsahoo/culture-evaluation{https://github.com/pramitsahoo/culture-evaluation}.