LAMAR：一種開放的語言感知多語言對齊重排序器

摘要

在多語言檢索增強生成中，檢索器可檢索多種語言的相關文檔，這些文檔在答案生成前會先經過重排序。然而，現有之多語言重排序器在排序語義相關的候選文檔時，是否考慮文檔語言仍不明確。我們的分析顯示，當跨語言存在語義等效的文檔時，這些重排序器並未一致地優先排序與查詢同語言的文檔，儘管文檔語言可能影響答案生成。我們釋出LAMAR，一個基於語言感知的多語言跨編碼器，其訓練目標為同時考量語義相關性與語言一致性。LAMAR首先採用英文錨定相關性蒸餾，以建立跨多語言輸入的一致相關性評分，隨後應用語言一致性的偏好對齊，促使與查詢同語言的文檔獲得更高排序，同時保留語義相關性。在針對語言一致性設計的對照實驗中，LAMAR在整體及所有個別語言上均達成最佳表現。LAMAR在既有之多語言重排序基準上亦保持競爭力。在實際檢索場景中，當對第一階段檢索出的候選文檔進行重排序時，LAMAR在所有報告指標上均達成最佳結果。這些成果顯示，LAMAR在一般多語言重排序基準上表現強勁之餘，亦考量了語言一致性。

English

In multilingual retrieval augmented generation, a retriever can retrieve relevant documents written in multiple languages, which are subsequently reranked before answer generation. However, it remains unclear whether existing multilingual rerankers consider document language when ordering semantically relevant candidates. Our analysis shows that these rerankers do not consistently prioritize documents written in the same language as the query when semantically equivalent documents are available across languages, even though document language can affect answer generation. We release LAMAR, a language aware multilingual cross encoder trained to account for both semantic relevance and language coherence. LAMAR first uses English anchored relevance distillation to establish consistent relevance scoring across multilingual inputs and then applies preference alignment for language coherence to encourage documents written in the same language as the query to receive higher rankings while retaining semantic relevance. In a controlled experiment designed to assess language coherence, LAMAR achieves the best performance overall and across all languages examined individually. LAMAR also remains competitive on established multilingual reranking benchmarks. In practical retrieval settings, LAMAR achieves the best results across all reported metrics when reranking candidates retrieved in the first stage. These results demonstrate that LAMAR accounts for language coherence while achieving strong performance on general multilingual reranking benchmarks.