GAPrune：面向领域感知嵌入的梯度对齐剪枝

GAPrune: Gradient-Alignment Pruning for Domain-Aware Embeddings

September 13, 2025

作者: Yixuan Tang, Yi Yang

cs.AI

摘要

特定領域的嵌入模型在需要專業語義理解的應用中展現出潛力，例如編碼代理和金融檢索系統，通常比通用模型獲得更高的性能提升。然而，最先進的嵌入模型通常基於包含數十億參數的大型語言模型（LLMs），這使得在資源受限的環境中部署變得具有挑戰性。通過剪枝進行模型壓縮提供了一個有前景的解決方案，但現有的剪枝方法均勻處理所有參數，未能區分通用語義表示和特定領域模式，導致次優的剪枝決策。因此，我們提出了GAPrune，這是一個剪枝框架，通過考慮領域重要性和保留通用語言基礎來應對這一挑戰。我們的方法使用費雪信息來衡量重要性，並通過通用領域梯度對齊來評估參數行為，然後使用我們的領域對齊重要性（DAI）評分來結合這些信號。較低的DAI分數表明該參數對領域任務的重要性較低，或在領域和通用目標之間產生衝突。在兩個領域基準測試（FinMTEB和ChemTEB）上的實驗表明，GAPrune在50%稀疏度的一次性剪枝中保持了與密集模型在2.5%以內的性能，同時優於所有基線。在100步的重新訓練中，GAPrune在FinMTEB上實現了+4.51%的提升，在ChemTEB上實現了+1.73%的提升，證明我們的剪枝策略不僅保留了特定領域的能力，還增強了這些能力。我們的研究結果表明，基於原則的剪枝策略可以實現模型壓縮和增強領域專業化，為研究社區提供了一種新的開發方法。

English

Domain-specific embedding models have shown promise for applications that require specialized semantic understanding, such as coding agents and financial retrieval systems, often achieving higher performance gains than general models. However, state-of-the-art embedding models are typically based on LLMs, which contain billions of parameters, making deployment challenging in resource-constrained environments. Model compression through pruning offers a promising solution, but existing pruning methods treat all parameters uniformly, failing to distinguish between general semantic representations and domain-specific patterns, leading to suboptimal pruning decisions. Thus, we propose GAPrune, a pruning framework that addresses this challenge by considering both domain importance and preserving general linguistic foundation. Our method uses Fisher Information to measure importance and general-domain gradient alignment to assess parameter behavior, then combines these signals using our Domain Alignment Importance (DAI) scoring. Lower DAI scores indicate that the parameter is either less important for the domain task or creates conflicts between domain and general objectives. Experiments on two domain benchmarks, FinMTEB and ChemTEB, show that GAPrune maintains performance within 2.5% of dense models in one-shot pruning at 50% sparsity, while outperforming all baselines. With retraining in 100 steps, GAPrune achieves +4.51% improvement on FinMTEB and +1.73% on ChemTEB, demonstrating that our pruning strategy not only preserves but enhances domain-specific capabilities. Our findings demonstrate that principled pruning strategies can achieve model compression and enhanced domain specialization, providing the research community with a new approach for development.