DoLa：透過對比層進行解碼，提高大型語言模型的事實性

摘要

儘管大型語言模型（LLMs）具有令人印象深刻的能力，但容易出現幻覺，即生成與預訓練期間觀察到的事實偏離的內容。我們提出了一種簡單的解碼策略，用於減少預訓練LLMs的幻覺，不需要條件設定在檢索的外部知識或額外的微調。我們的方法通過對比從將後期層與早期層投影到詞彙空間獲得的對數的差異，獲取下一令牌分佈，利用了在LLMs中事實知識通常被顯示為局部化於特定變壓器層的事實。我們發現，這種對比層解碼（DoLa）方法能夠更好地展現事實知識並減少不正確事實的生成。DoLa在多選任務和開放式生成任務中持續提高真實性，例如將LLaMA系列模型在TruthfulQA上的表現提高了12-17個絕對百分點，展示了其在使LLMs可靠生成真實事實方面的潛力。

English

Despite their impressive capabilities, large language models (LLMs) are prone to hallucinations, i.e., generating content that deviates from facts seen during pretraining. We propose a simple decoding strategy for reducing hallucinations with pretrained LLMs that does not require conditioning on retrieved external knowledge nor additional fine-tuning. Our approach obtains the next-token distribution by contrasting the differences in logits obtained from projecting the later layers versus earlier layers to the vocabulary space, exploiting the fact that factual knowledge in an LLMs has generally been shown to be localized to particular transformer layers. We find that this Decoding by Contrasting Layers (DoLa) approach is able to better surface factual knowledge and reduce the generation of incorrect facts. DoLa consistently improves the truthfulness across multiple choices tasks and open-ended generation tasks, for example improving the performance of LLaMA family models on TruthfulQA by 12-17% absolute points, demonstrating its potential in making LLMs reliably generate truthful facts.

DoLa：透過對比層進行解碼，提高大型語言模型的事實性

DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models

摘要

Support