ZipLoRA：透過有效合併LoRA實現任意主題與風格的融合

摘要

針對概念驅動個人化微調生成式模型的方法，通常在主題驅動或風格驅動的生成任務上能取得優異成果。近期提出的低秩自適應（LoRA）技術，成為實現概念驅動個人化的參數高效方案。雖然現有研究探索透過組合多個獨立LoRA模組來實現風格與主題的聯合生成，但現有技術尚無法穩定解決此問題，往往需在主題還原度或風格還原度之間做出妥協。我們提出ZipLoRA方法，能以低成本且高效的方式融合獨立訓練的風格與主題LoRA模組，從而實現任意使用者指定主題與風格的組合生成。透過對多樣化主題與風格組合的實驗驗證，ZipLoRA能在保持情境重構能力的同时，生成令人信服的結果，並在主題與風格還原度方面較基準方法實現顯著提升。專案頁面：https://ziplora.github.io

English

Methods for finetuning generative models for concept-driven personalization generally achieve strong results for subject-driven or style-driven generation. Recently, low-rank adaptations (LoRA) have been proposed as a parameter-efficient way of achieving concept-driven personalization. While recent work explores the combination of separate LoRAs to achieve joint generation of learned styles and subjects, existing techniques do not reliably address the problem; they often compromise either subject fidelity or style fidelity. We propose ZipLoRA, a method to cheaply and effectively merge independently trained style and subject LoRAs in order to achieve generation of any user-provided subject in any user-provided style. Experiments on a wide range of subject and style combinations show that ZipLoRA can generate compelling results with meaningful improvements over baselines in subject and style fidelity while preserving the ability to recontextualize. Project page: https://ziplora.github.io

ZipLoRA：透過有效合併LoRA實現任意主題與風格的融合

ZipLoRA: Any Subject in Any Style by Effectively Merging LoRAs

摘要

Support