LAMAR：一种开放的语言感知多语言对齐重排序器

摘要

在多语言检索增强生成中，检索器能够检索出以多种语言撰写的相关文档，这些文档随后在答案生成之前会经过重排序处理。然而，现有的多语言重排序器在对语义相关的候选文档进行排序时，是否考虑了文档语言仍不明确。我们的分析表明，当跨语言存在语义等价的文档时，这些重排序器并未始终优先将与查询语言相同的文档排在前面，尽管文档语言会影响答案的生成。我们发布了LAMAR——一种具备语言感知能力的多语言交叉编码器，其训练目标兼顾语义相关性与语言一致性。LAMAR首先通过英语锚定相关性蒸馏，建立跨多语言输入的一致相关性评分，随后应用偏好对齐实现语言一致性，促使与查询语言相同的文档在保留语义相关性的前提下获得更高的排序位置。在评估语言一致性的受控实验中，LAMAR在整体性能及所有被单独考察的语言上均取得了最优结果。此外，LAMAR在已有的多语言重排序基准测试中仍保持竞争力。在实际检索场景中，对首阶段检索到的候选文档进行重排序时，LAMAR在所有报告指标上均取得了最佳结果。这些结果表明，LAMAR在兼顾语言一致性的同时，在通用多语言重排序基准测试中展现出强劲性能。

English

In multilingual retrieval augmented generation, a retriever can retrieve relevant documents written in multiple languages, which are subsequently reranked before answer generation. However, it remains unclear whether existing multilingual rerankers consider document language when ordering semantically relevant candidates. Our analysis shows that these rerankers do not consistently prioritize documents written in the same language as the query when semantically equivalent documents are available across languages, even though document language can affect answer generation. We release LAMAR, a language aware multilingual cross encoder trained to account for both semantic relevance and language coherence. LAMAR first uses English anchored relevance distillation to establish consistent relevance scoring across multilingual inputs and then applies preference alignment for language coherence to encourage documents written in the same language as the query to receive higher rankings while retaining semantic relevance. In a controlled experiment designed to assess language coherence, LAMAR achieves the best performance overall and across all languages examined individually. LAMAR also remains competitive on established multilingual reranking benchmarks. In practical retrieval settings, LAMAR achieves the best results across all reported metrics when reranking candidates retrieved in the first stage. These results demonstrate that LAMAR accounts for language coherence while achieving strong performance on general multilingual reranking benchmarks.