使用RAG 和少樣本上下文學習進行證據支持的事實檢查與LLMs

Evidence-backed Fact Checking using RAG and Few-Shot In-Context Learning with LLMs

August 22, 2024

作者: Ronit Singhal, Pransh Patwa, Parth Patwa, Aman Chadha, Amitava Das

cs.AI

摘要

鑒於社交媒體上廣泛傳播的錯誤資訊，實施針對線上主張的事實核查機制至關重要。手動驗證每個主張極具挑戰性，凸顯了自動事實核查系統的必要性。本文介紹了我們設計的系統，旨在應對此問題。我們利用Averitec數據集來評估主張的真實性。除了真實性預測外，我們的系統還提供支持證據，這些證據是從數據集中提取的。我們開發了一個檢索和生成（RAG）流程，從知識庫中提取相關證據句子，然後將其與主張一起輸入到大型語言模型（LLM）進行分類。我們還評估了多個LLM的少樣本上下文學習（ICL）能力。我們的系統實現了0.33的「Averitec」得分，比基準線提高了22%。所有代碼將在https://github.com/ronit-singhal/evidence-backed-fact-checking-using-rag-and-few-shot-in-context-learning-with-llms 上提供。

English

Given the widespread dissemination of misinformation on social media, implementing fact-checking mechanisms for online claims is essential. Manually verifying every claim is highly challenging, underscoring the need for an automated fact-checking system. This paper presents our system designed to address this issue. We utilize the Averitec dataset to assess the veracity of claims. In addition to veracity prediction, our system provides supporting evidence, which is extracted from the dataset. We develop a Retrieve and Generate (RAG) pipeline to extract relevant evidence sentences from a knowledge base, which are then inputted along with the claim into a large language model (LLM) for classification. We also evaluate the few-shot In-Context Learning (ICL) capabilities of multiple LLMs. Our system achieves an 'Averitec' score of 0.33, which is a 22% absolute improvement over the baseline. All code will be made available on All code will be made available on https://github.com/ronit-singhal/evidence-backed-fact-checking-using-rag-and-few-shot-in-context-learning-with-llms.