SQS：基於稀疏量化子分佈的貝氏深度神經網路壓縮

摘要

壓縮大規模神經網路對於在資源受限裝置上部署模型至關重要。大多數現有方法分別採用權重剪枝或低位元量化，往往為了保持可接受的效能下降幅度，而導致壓縮率不理想。我們提出一個透過貝氏變分學習（\method）同時進行剪枝與低位元量化的統一框架，該框架在維持相當效能的同時，達到比先前基線更高的壓縮率。其關鍵概念是採用 spike-and-slab 先驗來誘導稀疏性，並使用高斯混合模型（GMMs）對量化權重建模，以實現低位元精度。由於結合 spike-and-slab 先驗與 GMMs 的目標函數難以計算，我們推導出一種高效近似，使有效壓縮得以實現，並將準確率損失降至最低。在理論上，我們為所提出用於稀疏且量化深度神經網路的變分方法提供了一致性結果。在壓縮 ResNet、BERT-base、Llama3.2 與 Qwen2.5 模型上的廣泛實驗顯示，我們的方法在效能下降幅度相當的情況下，達到比一系列現有方法更高的壓縮率。專案頁面：https://comeusr.github.io/SQS_Webpage。

English

Compressing large-scale neural networks is essential for deploying models on resource-constrained devices. Most existing methods adopt weight pruning or low-bit quantization individually, often resulting in suboptimal compression rates to preserve acceptable performance drops. We introduce a unified framework for simultaneous pruning and low-bit quantization via Bayesian variational learning (\method), which achieves higher compression rates than prior baselines while maintaining comparable performance. The key idea is to employ a spike-and-slab prior to induce sparsity and model quantized weights using Gaussian Mixture Models (GMMs) to enable low-bit precision. Due to the intractability of the objective involving spike-and-slab priors with GMMs, we derive an efficient approximation that facilitates effective compression with minimal accuracy loss. In theory, we provide a consistent result for our proposed variational approach to a sparse and quantized deep neural network. Extensive experiments on compressing ResNet, BERT-base, Llama3.2, and Qwen2.5 models show that our method achieves higher compression rates than a line of existing methods with comparable performance drops. Project page: https://comeusr.github.io/SQS_Webpage.