Llama-Mobile：高效VLMs的2.7位元量化

摘要

在行動裝置上部署視覺語言模型（VLMs）因其龐大的記憶體與運算需求而具有挑戰性。我們提出一個框架，用於對VLM進行量化，以在資源受限的硬體上實現高效推論。我們的方法結合了一條量化管線與一種新穎的2.7位元/參數格式；該管線利用模型本身生成訓練資料，且無需取得訓練設定，而此格式則支援在Arm CPU上高效執行。我們透過將Llama 3.2 11B Vision Instruct模型壓縮至3.7 GB並使用8位元激活值來驗證我們的方法，同時在一組標準視覺問答任務上保持強勁的效能。

English

Deploying vision-language models (VLMs) on mobile devices is challenging due to their significant memory and compute requirements. We present a framework for quantizing VLMs for efficient inference on resource-constrained hardware. Our approach combines a quantization pipeline that uses the model itself to generate training data and does not require access to the training setup, with a novel 2.7-bit-per-parameter format supporting efficient execution on Arm CPUs. We validate our approach by compressing the Llama 3.2 11B Vision Instruct model to 3.7 GB with 8-bit activations, preserving strong performance on a set of standard visual question answering tasks.