迷失在中間：語言模型如何使用長文本

Lost in the Middle: How Language Models Use Long Contexts

July 6, 2023

作者: Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang

cs.AI

摘要

儘管最近的語言模型具有接受長文本內容的能力，但對於語言模型如何有效利用更長的上下文仍知之甚少。我們分析語言模型在兩個需要識別其輸入上下文中相關信息的任務上的表現：多文件問答和鍵-值檢索。我們發現，當相關信息出現在輸入上下文的開頭或結尾時，表現通常最佳，但當模型必須訪問長上下文中的相關信息時，表現顯著下降。此外，隨著輸入上下文變得更長，即使對於明確設計為長上下文的模型，性能也會顯著降低。我們的分析有助於更好地理解語言模型如何使用其輸入上下文，並為未來長上下文模型提供新的評估協議。

English

While recent language models have the ability to take long contexts as input, relatively little is known about how well the language models use longer context. We analyze language model performance on two tasks that require identifying relevant information within their input contexts: multi-document question answering and key-value retrieval. We find that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts. Furthermore, performance substantially decreases as the input context grows longer, even for explicitly long-context models. Our analysis provides a better understanding of how language models use their input context and provides new evaluation protocols for future long-context models.