透過語義壓縮擴展大型語言模型的上下文窗口

摘要

基於Transformer的大型語言模型（LLMs）通常對文本輸入的長度進行限制，以確保生成流暢且相關的回應。這種限制限制了它們在涉及長文本的情境中的應用。我們提出了一種新穎的語義壓縮方法，使其能夠泛化到長度為原始文本的6-8倍的文本，而無需承擔顯著的計算成本或需要微調。我們提出的框架靈感來自信息理論中的源編碼，利用預訓練模型來減少長輸入的語義冗余，然後將其傳遞給LLMs進行下游任務。實驗結果表明，我們的方法有效地擴展了LLMs的上下文窗口，涵蓋問答、摘要、少樣本學習和信息檢索等一系列任務。此外，所提出的語義壓縮方法在減少相關計算開銷的同時，在文本生成中展現出一致的流暢性。

English

Transformer-based Large Language Models (LLMs) often impose limitations on the length of the text input to ensure the generation of fluent and relevant responses. This constraint restricts their applicability in scenarios involving long texts. We propose a novel semantic compression method that enables generalization to texts that are 6-8 times longer, without incurring significant computational costs or requiring fine-tuning. Our proposed framework draws inspiration from source coding in information theory and employs a pre-trained model to reduce the semantic redundancy of long inputs before passing them to the LLMs for downstream tasks. Experimental results demonstrate that our method effectively extends the context window of LLMs across a range of tasks including question answering, summarization, few-shot learning, and information retrieval. Furthermore, the proposed semantic compression method exhibits consistent fluency in text generation while reducing the associated computational overhead.

透過語義壓縮擴展大型語言模型的上下文窗口

Extending Context Window of Large Language Models via Semantic Compression

摘要

Support