風格偏見：少數人類視覺線索驅動多模態大型語言模型中的多數社會偏見

摘要

多模態大型語言模型（MLLMs）正日益被部署於對個人與社會影響重大的場景中，然而影響這些模型評判他人的視覺線索仍未被充分理解。先前的研究常比較不同（群體）個體，使得難以區分外貌效應與身份差異。我們提出StylisticBias，一個用於評估MLLMs中屬性層級社會偏見的受控基準。我們生成500張逼真的基礎臉孔，並為每張臉孔創造約50種單一屬性變體，總計約25,000張影像。此設計固定身份特徵，每次僅改變一項視覺屬性，使我們能衡量特定線索如何改變模型判斷。我們在25個二元社會判斷情境中評估六種MLLMs。結果顯示，年齡與體型主導身份層級效應，而時尚風格及其他視覺線索則引發最大的屬性層級變化。我們進一步發現，約15個屬性便佔據總變異量的近80%，顯示偏見集中於少數視覺線索。模型在與外貌語義相關的判斷，尤其是社會經濟地位與風格相關的判斷中，具有最強敏感性。我們釋出StylisticBias作為多模態模型細粒度偏見評估的基準。程式碼與資料集：https://github.com/timo-cavelius/StylisticBias 及 https://hf.co/datasets/shaghayegh/stylistic-bias-dataset。

English

Multimodal large language models (MLLMs) are increasingly deployed in personally and societally consequential settings, yet the visual cues that shape how these models judge people remain poorly understood. Prior work often compares different (groups of) individuals, making it difficult to separate appearance effects from identity differences. We introduce StylisticBias, a controlled benchmark for evaluating attribute-level social bias in MLLMs. We generate 500 photorealistic base faces and create about 50 single-attribute variations per face, producing about 25K images. This design keeps identity fixed and changes one visual attribute at a time. It lets us measure how specific cues shift model judgments. We evaluate six MLLMs across 25 binary social judgment scenarios. We find that age and body type dominate identity-level effects, while fashion style and other visual cues drive the largest attribute-level shifts. We further find that about 15 attributes account for nearly 80\% of the total variation, showing that bias is concentrated in a small set of visual cues. Sensitivity is strongest in judgments that are semantically aligned with appearance, especially socioeconomic and style-related judgments. We release StylisticBias as a benchmark for fine-grained bias evaluation in multimodal models. Code and dataset: https://github.com/timo-cavelius/StylisticBias and https://hf.co/datasets/shaghayegh/stylistic-bias-dataset.