利用RAG和少样本情境学习进行基于证据的事实核查与LLMs

Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMs

August 22, 2024

作者: Ronit Singhal, Pransh Patwa, Parth Patwa, Aman Chadha, Amitava Das

cs.AI

摘要

鉴于社交媒体上虚假信息的广泛传播，实施针对在线声明的事实核查机制至关重要。手动验证每一项声明具有极高的挑战性，突显了自动事实核查系统的必要性。本文介绍了我们设计的旨在解决这一问题的系统。我们利用Averitec数据集评估声明的真实性。除了真实性预测外，我们的系统还提供从数据集中提取的支持证据。我们开发了一个检索和生成（RAG）流程，从知识库中提取相关证据句子，然后将其与声明一起输入到大型语言模型（LLM）进行分类。我们还评估了多个LLM的少样本上下文学习（ICL）能力。我们的系统实现了0.33的“Averitec”得分，比基准线提高了22%。所有代码将在https://github.com/ronit-singhal/evidence-backed-fact-checking-using-rag-and-few-shot-in-context-learning-with-llms 上提供。

English

Given the widespread dissemination of misinformation on social media, implementing fact-checking mechanisms for online claims is essential. Manually verifying every claim is highly challenging, underscoring the need for an automated fact-checking system. This paper presents our system designed to address this issue. We utilize the Averitec dataset to assess the veracity of claims. In addition to veracity prediction, our system provides supporting evidence, which is extracted from the dataset. We develop a Retrieve and Generate (RAG) pipeline to extract relevant evidence sentences from a knowledge base, which are then inputted along with the claim into a large language model (LLM) for classification. We also evaluate the few-shot In-Context Learning (ICL) capabilities of multiple LLMs. Our system achieves an 'Averitec' score of 0.33, which is a 22% absolute improvement over the baseline. All code will be made available on All code will be made available on https://github.com/ronit-singhal/evidence-backed-fact-checking-using-rag-and-few-shot-in-context-learning-with-llms.