解碼壓縮信任：審視壓縮下高效LLM的可信度

Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression

March 18, 2024

作者: Junyuan Hong, Jinhao Duan, Chenhui Zhang, Zhangheng Li, Chulin Xie, Kelsey Lieberman, James Diffenderfer, Brian Bartoldson, Ajay Jaiswal, Kaidi Xu, Bhavya Kailkhura, Dan Hendrycks, Dawn Song, Zhangyang Wang, Bo Li

cs.AI

摘要

對於資源高效的推論，壓縮高性能大型語言模型（LLMs）已成為一種受歡迎的策略。雖然最先進的壓縮方法在保留良性任務表現方面取得了令人印象深刻的進展，但在安全性和可信度方面的潛在風險卻大多被忽略。本研究首次對三（3）個領先的LLMs進行了全面評估，使用了五（5）種最先進的壓縮技術，跨越八（8）個可信度維度。我們的實驗凸顯了壓縮與可信度之間的複雜相互作用，揭示了一些有趣的模式。我們發現，在當前情況下，量化比修剪更有效，可以同時實現效率和可信度。例如，一個4位量化模型保留了其原始對應物的可信度，但模型修剪明顯降低了可信度，即使在50％的稀疏度下也是如此。此外，在適中的位範圍內使用量化可能會意外地提高某些可信度維度，如倫理和公平性。相反，對非常低位級（3位）的極端量化往往會顯著降低可信度。僅通過觀察良性表現無法揭示這種增加的風險，因此實踐中需要進行全面的可信度評估。這些發現為在LLMs中同時實現高效用、效率和可信度提供了實用建議。模型和代碼可在https://decoding-comp-trust.github.io/找到。

English

Compressing high-capability Large Language Models (LLMs) has emerged as a favored strategy for resource-efficient inferences. While state-of-the-art (SoTA) compression methods boast impressive advancements in preserving benign task performance, the potential risks of compression in terms of safety and trustworthiness have been largely neglected. This study conducts the first, thorough evaluation of three (3) leading LLMs using five (5) SoTA compression techniques across eight (8) trustworthiness dimensions. Our experiments highlight the intricate interplay between compression and trustworthiness, revealing some interesting patterns. We find that quantization is currently a more effective approach than pruning in achieving efficiency and trustworthiness simultaneously. For instance, a 4-bit quantized model retains the trustworthiness of its original counterpart, but model pruning significantly degrades trustworthiness, even at 50% sparsity. Moreover, employing quantization within a moderate bit range could unexpectedly improve certain trustworthiness dimensions such as ethics and fairness. Conversely, extreme quantization to very low bit levels (3 bits) tends to significantly reduce trustworthiness. This increased risk cannot be uncovered by looking at benign performance alone, in turn, mandating comprehensive trustworthiness evaluation in practice. These findings culminate in practical recommendations for simultaneously achieving high utility, efficiency, and trustworthiness in LLMs. Models and code are available at https://decoding-comp-trust.github.io/.