GLIMMER:通用後期交互記憶重新排序器
GLIMMER: generalized late-interaction memory reranker
June 17, 2023
作者: Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Sumit Sanghai, William W. Cohen, Joshua Ainslie
cs.AI
摘要
記憶增強是一種有效將外部資訊納入語言模型的強大方法,但相對於檢索文本會導致性能降低。最近的研究引入了LUMEN,一種記憶檢索混合方法,部分預先計算記憶並使用較小的即時編碼器動態更新記憶表示。
我們提出GLIMMER,通過以下方式改進這種方法:1) 利用強大記憶表示的免費訪問,通過在記憶頂部應用淺層重新排序器,以極大降低成本大幅改善檢索質量,2) 結合多任務訓練,學習一個通用且高質量的記憶和即時編碼器。GLIMMER在知識密集型任務的KILT基準測試中相比於LUMEN和FiD實現了性能的顯著提升並且速度更快。
English
Memory-augmentation is a powerful approach for efficiently incorporating
external information into language models, but leads to reduced performance
relative to retrieving text. Recent work introduced LUMEN, a memory-retrieval
hybrid that partially pre-computes memory and updates memory representations on
the fly with a smaller live encoder.
We propose GLIMMER, which improves on this approach through 1) exploiting
free access to the powerful memory representations by applying a shallow
reranker on top of memory to drastically improve retrieval quality at low cost,
and 2) incorporating multi-task training to learn a general and higher quality
memory and live encoder. GLIMMER achieves strong gains in performance at faster
speeds compared to LUMEN and FiD on the KILT benchmark of knowledge-intensive
tasks.