B-score：基于响应历史检测大规模语言模型中的偏见

B-score: Detecting biases in large language models using response history

May 24, 2025

作者: An Vo, Mohammad Reza Taesiri, Daeyoung Kim, Anh Totti Nguyen

cs.AI

摘要

大型语言模型（LLMs）常表现出显著的偏见，例如对女性的偏见或对数字7的偏好。我们探讨了在多轮对话中，当LLMs能够观察到自己对同一问题的先前回答时，是否能够输出偏见较少的答案。为了理解哪些类型的问题更容易引发偏见性回答，我们在提出的涵盖9个主题、分为三种类型的问题集上测试了LLMs：（1）主观性；（2）随机性；以及（3）客观性。有趣的是，在多轮对话中，面对寻求随机、无偏见答案的问题时，LLMs能够实现“自我去偏”。此外，我们提出了B-score这一新颖指标，它在检测对主观性、随机性、简单及困难问题的偏见方面表现有效。在MMLU、HLE和CSQA数据集上，相较于仅使用语言化置信度分数或单轮回答频率，利用B-score显著提升了LLM答案的验证准确率（即接受LLM的正确答案并拒绝错误答案）。代码和数据可在以下网址获取：https://b-score.github.io。

English

Large language models (LLMs) often exhibit strong biases, e.g, against women or in favor of the number 7. We investigate whether LLMs would be able to output less biased answers when allowed to observe their prior answers to the same question in a multi-turn conversation. To understand which types of questions invite more biased answers, we test LLMs on our proposed set of questions that span 9 topics and belong to three types: (1) Subjective; (2) Random; and (3) Objective. Interestingly, LLMs are able to "de-bias" themselves in a multi-turn conversation in response to questions that seek an Random, unbiased answer. Furthermore, we propose B-score, a novel metric that is effective in detecting biases to Subjective, Random, Easy, and Hard questions. On MMLU, HLE, and CSQA, leveraging B-score substantially improves the verification accuracy of LLM answers (i.e, accepting LLM correct answers and rejecting incorrect ones) compared to using verbalized confidence scores or the frequency of single-turn answers alone. Code and data are available at: https://b-score.github.io.