GLIMMER：通用后交互记忆重排序器

GLIMMER: generalized late-interaction memory reranker

June 17, 2023

作者: Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Sumit Sanghai, William W. Cohen, Joshua Ainslie

cs.AI

摘要

记忆增强是一种强大的方法，可以高效地将外部信息整合到语言模型中，但相对于检索文本会导致性能下降。最近的研究引入了LUMEN，这是一种记忆检索混合方法，部分预先计算记忆，并通过较小的实时编码器实时更新记忆表示。 我们提出了GLIMMER，通过以下方式改进这种方法：1）利用对强大记忆表示的免费访问，通过在记忆之上应用浅层重新排序器，大幅提高检索质量，成本较低；2）结合多任务训练，学习通用和更高质量的记忆和实时编码器。GLIMMER在KILT知识密集型任务基准测试中相比LUMEN和FiD，以更快的速度取得了性能上的显著增益。

English

Memory-augmentation is a powerful approach for efficiently incorporating external information into language models, but leads to reduced performance relative to retrieving text. Recent work introduced LUMEN, a memory-retrieval hybrid that partially pre-computes memory and updates memory representations on the fly with a smaller live encoder. We propose GLIMMER, which improves on this approach through 1) exploiting free access to the powerful memory representations by applying a shallow reranker on top of memory to drastically improve retrieval quality at low cost, and 2) incorporating multi-task training to learn a general and higher quality memory and live encoder. GLIMMER achieves strong gains in performance at faster speeds compared to LUMEN and FiD on the KILT benchmark of knowledge-intensive tasks.