对齐中的语言链：跨语言排序偏好优化

摘要

大语言模型的对齐高度依赖于以英语为中心的高质量偏好数据，这往往导致模型在其他语言中的性能欠佳。本文提出了跨语言排序偏好优化（CRPO）框架，该框架利用来自英语的稳健偏好知识来促进目标语言中的偏好对齐。我们在目标语言与英语之间的并行偏好对中设计了层级结构，以联合优化语内与语间偏好，从而增强语言适配性和输出质量。CRPO基于LambdaLoss框架，通过提供多个候选响应之间的相对排序信号，超越了基于二元比较的优化方式。我们在五种资源规模各异的语言上进行的实验表明，CRPO在指令遵循能力和知识利用能力方面均持续优于标准方法。值得注意的是，在不同权重方案下观察到的稳健性能提升进一步验证了我们的层级设计在多语言设置中的实证有效性。此外，我们的研究结果强调，CRPO显著提升了奖励边际和理想响应的对数概率，为跨语言对齐贡献了更稳定的偏好流形。我们的代码可在https://github.com/dltmddbs100/CRPO获取。

English

The alignment of Large Language Models heavily relies on English-centric high-quality preference data, which often leads to suboptimal performance in other languages. In this paper, we propose Cross-lingual Ranking Preference Optimization~(CRPO), a novel framework that leverages robust preference knowledge from English to facilitate preference alignment in the target language. We design a hierarchical structure within parallel preference pairs across the target language and English to jointly optimize intra- and inter-lingual preferences, thereby enhancing language adaptation and output quality. Building on the LambdaLoss framework, CRPO goes beyond the binary comparison based optimization by providing a relative ranking signal across multiple candidate responses. Our experiments across five languages with varying resource scales demonstrate that CRPO consistently outperforms standard approaches in both instruction-following and knowledge utilization capability. Notably, the robust performance gains observed across various weighting schemes further validate the empirical effectiveness of our hierarchical design in a multilingual setup. Furthermore, our findings highlight that CRPO significantly improves both reward margins and the log-probability of desirable responses, contributing to a more stable preference manifold for cross-lingual alignment. Our code is available at https://github.com/dltmddbs100/CRPO.