COMPOT：面向變壓器壓縮的校準優化矩陣普羅克魯斯正交化方法

摘要

變壓器模型的訓練後壓縮通常依賴截斷奇異值分解（SVD）。然而，強制使用單一共享子空間即使在中等壓縮程度下也可能導致準確度下降。稀疏字典學習提供了更靈活的子空間聯合表示，但現有方法常受制於迭代式的字典與係數更新。我們提出COMPOT（針對變壓器的校準優化矩陣普羅克魯斯正交化），這是一種免訓練的壓縮框架，利用小型校準數據集來估計稀疏權重分解。COMPOT採用正交字典，可對字典進行閉式普羅克魯斯更新，並對係數進行解析型單步稀疏編碼，從而消除迭代優化過程。為應對全局壓縮預算下的異質層級敏感度問題，COMPOT進一步引入一次性動態分配策略，自適應地重新分配各層壓縮率。在多樣化架構與任務上的大量實驗表明，COMPOT相較於強勁的低秩與稀疏基準方法，始終提供更優的質量-壓縮權衡，同時能完全兼容訓練後量化以實現極致壓縮。程式碼可於此處取得：https://github.com/mts-ai/COMPOT。

English

Post-training compression of Transformer models commonly relies on truncated singular value decomposition (SVD). However, enforcing a single shared subspace can degrade accuracy even at moderate compression. Sparse dictionary learning provides a more flexible union-of-subspaces representation, but existing approaches often suffer from iterative dictionary and coefficient updates. We propose COMPOT (Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers), a training-free compression framework that uses a small calibration dataset to estimate a sparse weight factorization. COMPOT employs orthogonal dictionaries that enable closed-form Procrustes updates for the dictionary and analytical single-step sparse coding for the coefficients, eliminating iterative optimization. To handle heterogeneous layer sensitivity under a global compression budget, COMPOT further introduces a one-shot dynamic allocation strategy that adaptively redistributes layer-wise compression rates. Extensive experiments across diverse architectures and tasks show that COMPOT consistently delivers a superior quality-compression trade-off over strong low-rank and sparse baselines, while remaining fully compatible with post-training quantization for extreme compression. Code is available https://github.com/mts-ai/COMPOT{here}.

COMPOT：面向變壓器壓縮的校準優化矩陣普羅克魯斯正交化方法

COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression

摘要

Support