無 GPT 排名：在開源大型語言模型上構建獨立於 GPT 的列表式重新排列器

摘要

基於大型語言模型（LLM）的Listwise重新排序器是零樣本最先進的技術。然而，目前在這個方向上的作品都依賴於GPT模型，這使得科學可重現性存在單一失敗點。此外，這引發了一個擔憂，即目前的研究結果僅適用於GPT模型，而不適用於LLM整體。在這項工作中，我們解除了這個先決條件，首次建立了在不依賴於GPT的情況下具有效果的Listwise重新排序器。我們的段落檢索實驗表明，我們最佳的Listwise重新排序器超越了基於GPT-3.5的Listwise重新排序器13％，並實現了相當於基於GPT-4所建立的97％效果。我們的結果還表明，現有的訓練數據集，這些數據集明確為點對點排序而構建，不足以建立這樣的Listwise重新排序器。相反，需要高質量的Listwise排序數據，這是必不可少的，需要進一步努力建立人工標註的Listwise數據資源。

English

Listwise rerankers based on large language models (LLM) are the zero-shot state-of-the-art. However, current works in this direction all depend on the GPT models, making it a single point of failure in scientific reproducibility. Moreover, it raises the concern that the current research findings only hold for GPT models but not LLM in general. In this work, we lift this pre-condition and build for the first time effective listwise rerankers without any form of dependency on GPT. Our passage retrieval experiments show that our best list se reranker surpasses the listwise rerankers based on GPT-3.5 by 13% and achieves 97% effectiveness of the ones built on GPT-4. Our results also show that the existing training datasets, which were expressly constructed for pointwise ranking, are insufficient for building such listwise rerankers. Instead, high-quality listwise ranking data is required and crucial, calling for further work on building human-annotated listwise data resources.

無 GPT 排名：在開源大型語言模型上構建獨立於 GPT 的列表式重新排列器

Rank-without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Models

摘要

Support