通过语义压缩扩展大型语言模型的上下文窗口

摘要

基于Transformer的大型语言模型（LLMs）通常会对文本输入长度施加限制，以确保生成流畅且相关的回复。这种约束限制了它们在涉及长文本的场景中的适用性。我们提出了一种新颖的语义压缩方法，可以使模型泛化到长文本的情况，而不会带来显著的计算成本或需要微调。我们提出的框架从信息论中的源编码中汲取灵感，利用预训练模型来减少长输入的语义冗余，然后将其传递给LLMs用于下游任务。实验结果表明，我们的方法有效地扩展了LLMs的上下文窗口，涵盖了一系列任务，包括问答、摘要、少样本学习和信息检索。此外，所提出的语义压缩方法在减少相关计算开销的同时，在文本生成中表现出一致的流畅性。

English

Transformer-based Large Language Models (LLMs) often impose limitations on the length of the text input to ensure the generation of fluent and relevant responses. This constraint restricts their applicability in scenarios involving long texts. We propose a novel semantic compression method that enables generalization to texts that are 6-8 times longer, without incurring significant computational costs or requiring fine-tuning. Our proposed framework draws inspiration from source coding in information theory and employs a pre-trained model to reduce the semantic redundancy of long inputs before passing them to the LLMs for downstream tasks. Experimental results demonstrate that our method effectively extends the context window of LLMs across a range of tasks including question answering, summarization, few-shot learning, and information retrieval. Furthermore, the proposed semantic compression method exhibits consistent fluency in text generation while reducing the associated computational overhead.

通过语义压缩扩展大型语言模型的上下文窗口

Extending Context Window of Large Language Models via Semantic Compression

摘要

Support