EMLoC：基於模擬器的記憶體高效微調與LoRA校正

摘要

開源基礎模型已迅速獲得廣泛採用與發展，賦能於多樣化領域中的強大通用能力。然而，針對特定領域或個性化任務對大型基礎模型進行微調，由於其顯著的內存開銷遠超推理所需，對大多數用戶而言仍成本過高。我們提出了EMLoC，一種基於模擬器的內存高效微調框架，結合LoRA校正技術，使得模型微調能在與推理相同的內存預算內完成。EMLoC利用激活感知的奇異值分解（SVD）在小型下游校準集上構建任務特定的輕量級模擬器。隨後，通過LoRA在此輕量級模擬器上進行微調。為解決原始模型與壓縮模擬器之間的不對齊問題，我們提出了一種新穎的補償算法來校正微調後的LoRA模塊，從而可將其合併回原始模型用於推理。EMLoC支持靈活的壓縮比和標準訓練流程，使其能適應廣泛的應用場景。大量實驗表明，EMLoC在多個數據集和模態上均優於其他基線方法。此外，無需量化，EMLoC便能在單塊24GB消費級GPU上實現對380億參數模型的微調，為個體用戶帶來了高效且實用的模型適應方案。

English

Open-source foundation models have seen rapid adoption and development, enabling powerful general-purpose capabilities across diverse domains. However, fine-tuning large foundation models for domain-specific or personalized tasks remains prohibitively expensive for most users due to the significant memory overhead beyond that of inference. We introduce EMLoC, an Emulator-based Memory-efficient fine-tuning framework with LoRA Correction, which enables model fine-tuning within the same memory budget required for inference. EMLoC constructs a task-specific light-weight emulator using activation-aware singular value decomposition (SVD) on a small downstream calibration set. Fine-tuning then is performed on this lightweight emulator via LoRA. To tackle the misalignment between the original model and the compressed emulator, we propose a novel compensation algorithm to correct the fine-tuned LoRA module, which thus can be merged into the original model for inference. EMLoC supports flexible compression ratios and standard training pipelines, making it adaptable to a wide range of applications. Extensive experiments demonstrate that EMLoC outperforms other baselines across multiple datasets and modalities. Moreover, without quantization, EMLoC enables fine-tuning of a 38B model on a single 24GB consumer GPU-bringing efficient and practical model adaptation to individual users.

EMLoC：基於模擬器的記憶體高效微調與LoRA校正

EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correction

摘要

Support