无需 GPT 的排名：在开源大型语言模型上构建独立于 GPT 的列表重排序器

摘要

基于大型语言模型（LLM）的基于列表的重新排序器是零-shot 最先进的。然而，这一方向的当前工作都依赖于GPT模型，使得科学可重现性存在单一故障点。此外，这引发了一个担忧，即当前的研究结果仅适用于GPT模型，而不适用于LLM。在这项工作中，我们消除了这一先决条件，首次构建了在不依赖于GPT的情况下有效的基于列表的重新排序器。我们的段落检索实验表明，我们最佳的列表重新排序器比基于GPT-3.5的列表重新排序器高出13%，并且达到了基于GPT-4构建的列表重新排序器效果的97%。我们的结果还表明，现有的训练数据集，这些数据集明确是为点对点排序而构建的，不足以构建这种基于列表的重新排序器。相反，需要高质量的基于列表的排序数据，这是必不可少的，呼吁进一步努力构建人工注释的基于列表的数据资源。

English

Listwise rerankers based on large language models (LLM) are the zero-shot state-of-the-art. However, current works in this direction all depend on the GPT models, making it a single point of failure in scientific reproducibility. Moreover, it raises the concern that the current research findings only hold for GPT models but not LLM in general. In this work, we lift this pre-condition and build for the first time effective listwise rerankers without any form of dependency on GPT. Our passage retrieval experiments show that our best list se reranker surpasses the listwise rerankers based on GPT-3.5 by 13% and achieves 97% effectiveness of the ones built on GPT-4. Our results also show that the existing training datasets, which were expressly constructed for pointwise ranking, are insufficient for building such listwise rerankers. Instead, high-quality listwise ranking data is required and crucial, calling for further work on building human-annotated listwise data resources.

无需 GPT 的排名：在开源大型语言模型上构建独立于 GPT 的列表重排序器

Rank-without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Models

摘要

Support