利用大语言模型与人类专家事实核查方法对新闻媒体的真实性与偏见进行剖析

Profiling News Media for Factuality and Bias Using LLMs and the Fact-Checking Methodology of Human Experts

June 14, 2025

作者: Zain Muhammad Mujahid, Dilshod Azizov, Maha Tufail Agro, Preslav Nakov

cs.AI

摘要

在当今网络错误与虚假信息泛滥的时代，赋能读者理解所阅内容至关重要。此方向的重要努力依赖于人工或自动的事实核查，这对于信息有限的新兴主张而言颇具挑战。此类情境可通过评估信息来源的可靠性与政治倾向来处理，即对整个新闻机构而非单一主张或文章进行特征刻画。这是一个重要但研究尚不充分的方向。尽管先前工作已探讨了语言和社会背景，我们并不分析社交媒体中的个别文章或信息。相反，我们提出了一种新颖的方法论，模拟专业事实核查员评估整个媒体机构事实性与政治偏见的准则。具体而言，我们基于这些准则设计了多种提示，并引导大型语言模型（LLMs）生成响应，进而汇总这些响应以作出预测。除了通过多项LLMs的广泛实验展示相对于强基线模型的显著改进外，我们还深入分析了媒体流行度与地区对模型性能的影响。此外，我们进行了消融研究，以突出数据集中促成这些改进的关键组成部分。为促进未来研究，我们在https://github.com/mbzuai-nlp/llm-media-profiling 发布了数据集与代码。

English

In an age characterized by the proliferation of mis- and disinformation online, it is critical to empower readers to understand the content they are reading. Important efforts in this direction rely on manual or automatic fact-checking, which can be challenging for emerging claims with limited information. Such scenarios can be handled by assessing the reliability and the political bias of the source of the claim, i.e., characterizing entire news outlets rather than individual claims or articles. This is an important but understudied research direction. While prior work has looked into linguistic and social contexts, we do not analyze individual articles or information in social media. Instead, we propose a novel methodology that emulates the criteria that professional fact-checkers use to assess the factuality and political bias of an entire outlet. Specifically, we design a variety of prompts based on these criteria and elicit responses from large language models (LLMs), which we aggregate to make predictions. In addition to demonstrating sizable improvements over strong baselines via extensive experiments with multiple LLMs, we provide an in-depth error analysis of the effect of media popularity and region on model performance. Further, we conduct an ablation study to highlight the key components of our dataset that contribute to these improvements. To facilitate future research, we released our dataset and code at https://github.com/mbzuai-nlp/llm-media-profiling.