語言鏈在對齊中的應用：跨語言排名偏好優化

摘要

大型語言模型的對齊高度依賴以英語為中心的高品質偏好資料，這往往導致在其他語言上的表現次佳。在本文中，我們提出跨語言排序偏好最佳化（Cross-lingual Ranking Preference Optimization, CRPO），一個新穎的框架，利用來自英語的穩健偏好知識，以促進目標語言中的偏好對齊。我們在目標語言與英語的平行偏好對中設計了一種階層式結構，共同最佳化語言內與跨語言的偏好，從而增強語言適應與輸出品質。基於 LambdaLoss 框架，CRPO 超越了基於二元比較的最佳化，透過跨多個候選回應提供相對排序訊號。我們在五種資源規模不同的語言上進行的實驗顯示，CRPO 在指令遵循與知識利用能力上均一致地優於標準方法。值得注意的是，在不同加權方案下觀察到的穩健效能增益，進一步驗證了我們階層式設計在多語言設定中的實證有效性。此外，我們的發現強調，CRPO 顯著改善了獎勵邊際與理想回應的對數機率，有助於形成更穩定的偏好流形以實現跨語言對齊。我們的程式碼可在 https://github.com/dltmddbs100/CRPO 取得。

English

The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in other languages. In this paper, we propose Cross-lingual Ranking Preference Optimization~(CRPO), a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language. We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences, thereby enhancing language adaptation and output quality. Building on the LambdaLoss framework, CRPO goes beyond the binary comparison based optimization by providing a relative ranking signal across multiple candidate responses. Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization capability. Notably, the robust performance gains observed across various weighting schemes further validate the empirical effectiveness of our hierarchical design in a multilingual setup. Furthermore, our findings highlight that CRPO significantly improves both reward margins and the log-probability of desirable responses, contributing to a more stable preference manifold for cross-lingual alignment. Our code is available at https://github.com/dltmddbs100/CRPO.