StylisticBias: 少数人类视觉线索驱动多模态大语言模型中的大多数社会偏见

摘要

多模态大语言模型（MLLMs）正日益部署于对个人与社会具有重要影响的场景中，然而塑造这些模型如何评判他人的视觉线索仍知之甚少。以往研究常比较不同（群组）个体，导致难以将外貌效应从身份差异中分离。我们提出StylisticBias，一个用于评估MLLMs中属性层级社会偏见的受控基准。我们生成500张逼真基础人脸，并为每张人脸创建约50种单属性变体，总计约25,000张图像。该设计保持身份固定，每次仅改变一项视觉属性，使我们能够衡量特定线索如何改变模型判断。我们在25个二元社会判断场景中评估了六种MLLMs。研究发现，年龄和体型主导了身份层级的效应，而时尚风格及其他视觉线索则引发最大的属性层级偏移。进一步发现，约15个属性解释了近80%的总变异，表明偏见集中于少量视觉线索。敏感性在与外貌语义对齐的判断中最强，尤其是社会经济和风格相关判断。我们发布StylisticBias作为多模态模型细粒度偏见评估的基准。代码与数据集链接：https://github.com/timo-cavelius/StylisticBias 及 https://hf.co/datasets/shaghayegh/stylistic-bias-dataset。

English

Multimodal large language models (MLLMs) are increasingly deployed in personally and societally consequential settings, yet the visual cues that shape how these models judge people remain poorly understood. Prior work often compares different (groups of) individuals, making it difficult to separate appearance effects from identity differences. We introduce StylisticBias, a controlled benchmark for evaluating attribute-level social bias in MLLMs. We generate 500 photorealistic base faces and create about 50 single-attribute variations per face, producing about 25K images. This design keeps identity fixed and changes one visual attribute at a time. It lets us measure how specific cues shift model judgments. We evaluate six MLLMs across 25 binary social judgment scenarios. We find that age and body type dominate identity-level effects, while fashion style and other visual cues drive the largest attribute-level shifts. We further find that about 15 attributes account for nearly 80\% of the total variation, showing that bias is concentrated in a small set of visual cues. Sensitivity is strongest in judgments that are semantically aligned with appearance, especially socioeconomic and style-related judgments. We release StylisticBias as a benchmark for fine-grained bias evaluation in multimodal models. Code and dataset: https://github.com/timo-cavelius/StylisticBias and https://hf.co/datasets/shaghayegh/stylistic-bias-dataset.