COMPASS：基于自适应语义采样的持续多语言参数高效微调

COMPASS: COntinual Multilingual PEFT with Adaptive Semantic Sampling

April 22, 2026

作者: Noah Flynn

cs.AI

摘要

大型语言模型（LLMs）在不同语言间常存在性能差异，而简单的多语言微调往往会因负向跨语言干扰导致性能下降。为解决此问题，我们提出COMPASS（基于自适应语义采样的持续多语言参数高效微调框架），这是一种面向目标语言适配的数据驱动创新框架。该方法通过参数高效微调（PEFT）技术，在精挑细选的辅助多语言数据子集上训练轻量级语言专属适配器。其核心在于采用分布感知采样策略：利用多语言嵌入向量和聚类技术识别现有训练数据与目标使用分布之间的语义鸿沟。通过优先选取低表征度语义簇的辅助数据，COMPASS能最大化正向跨语言迁移效应，同时最小化干扰。我们进一步将其扩展为持续学习框架COMPASS-ECDA，该框架可监测生产环境中的数据分布漂移，动态更新适配器以防止模型老化，实现新数据适应与既有知识保存的平衡。在三种不同模型架构（Phi-4-Mini、Llama-3.1-8B和Qwen2.5-7B）及多个高难度多语言基准测试（Global-MMLU、MMLU-ProX）——包括未见过的长上下文任务（OneRuler）上的实验表明，COMPASS始终优于基于语言相似度的基线方法，为动态环境中高性能多语言模型的开发与维护提供了高效、可持续的解决方案。

English

Large language models (LLMs) often exhibit performance disparities across languages, with naive multilingual fine-tuning frequently degrading performance due to negative cross-lingual interference. To address this, we introduce COMPASS (COntinual Multilingual PEFT with Adaptive Semantic Sampling), a novel data-centric framework for adapting LLMs to target languages. COMPASS leverages parameter-efficient fine-tuning (PEFT) by training lightweight, language-specific adapters on a judiciously selected subset of auxiliary multilingual data. The core of our method is a distribution-aware sampling strategy that uses multilingual embeddings and clustering to identify semantic gaps between existing training data and a target usage distribution. By prioritizing auxiliary data from under-represented semantic clusters, COMPASS maximizes positive cross-lingual transfer while minimizing interference. We extend this into a continual learning framework, COMPASS-ECDA, which monitors for data distribution shifts in production and dynamically updates adapters to prevent model staleness, balancing adaptation to new data with the preservation of existing knowledge. Across three different model architectures (Phi-4-Mini, Llama-3.1-8B, and Qwen2.5-7B) and multiple challenging multilingual benchmarks (Global-MMLU, MMLU-ProX), including unseen long-context tasks (OneRuler), we demonstrate that COMPASS consistently outperforms baseline methods guided by linguistic similarity, providing an effective, efficient, and sustainable solution for developing and maintaining high-performing multilingual models in dynamic environments.