一維適配器統御全局：概念、擴散模型和消除應用

摘要

商業和開源擴散模型（DMs）在文本到圖像生成中的普遍應用促使風險管理以防止不良行為。學術界現有的概念擦除方法都基於完整參數或基於規範的微調，我們觀察到以下問題：1）生成變化朝向侵蝕：目標消除期間的參數漂移導致變化和潛在變形跨越所有生成，甚至在多概念消除時更明顯地侵蝕其他概念；2）轉移能力不足和部署效率低：先前的模型特定擦除阻礙了概念的靈活組合和向其他模型的無需訓練轉移，導致隨著部署場景增加成本呈線性增長。為實現非侵入性、精確、可定制和可轉移的擦除，我們將我們的擦除框架基於一維適配器，一次從大多數DMs中擦除多個概念，適用於多種擦除應用。將概念半透膜結構注入為膜（SPM）到任何DM中以學習有針對性的擦除，同時通過新穎的潛在錨定微調策略有效地緩解變化和侵蝕現象。一旦獲得，SPMs可以靈活組合並即插即用於其他DMs，無需特定的重新調整，實現對各種情景的及時和高效適應。在生成過程中，我們的促進傳輸機制動態調節每個SPM的滲透性以應對不同的輸入提示，進一步減少對其他概念的影響。在約40個概念、7個DM和4個擦除應用中的定量和定性結果已經證明了SPM的優越擦除效果。我們的代碼和預調整的SPMs將在項目頁面https://lyumengyao.github.io/projects/spm 上提供。

English

The prevalent use of commercial and open-source diffusion models (DMs) for text-to-image generation prompts risk mitigation to prevent undesired behaviors. Existing concept erasing methods in academia are all based on full parameter or specification-based fine-tuning, from which we observe the following issues: 1) Generation alternation towards erosion: Parameter drift during target elimination causes alternations and potential deformations across all generations, even eroding other concepts at varying degrees, which is more evident with multi-concept erased; 2) Transfer inability & deployment inefficiency: Previous model-specific erasure impedes the flexible combination of concepts and the training-free transfer towards other models, resulting in linear cost growth as the deployment scenarios increase. To achieve non-invasive, precise, customizable, and transferable elimination, we ground our erasing framework on one-dimensional adapters to erase multiple concepts from most DMs at once across versatile erasing applications. The concept-SemiPermeable structure is injected as a Membrane (SPM) into any DM to learn targeted erasing, and meantime the alteration and erosion phenomenon is effectively mitigated via a novel Latent Anchoring fine-tuning strategy. Once obtained, SPMs can be flexibly combined and plug-and-play for other DMs without specific re-tuning, enabling timely and efficient adaptation to diverse scenarios. During generation, our Facilitated Transport mechanism dynamically regulates the permeability of each SPM to respond to different input prompts, further minimizing the impact on other concepts. Quantitative and qualitative results across ~40 concepts, 7 DMs and 4 erasing applications have demonstrated the superior erasing of SPM. Our code and pre-tuned SPMs will be available on the project page https://lyumengyao.github.io/projects/spm.

一維適配器統御全局：概念、擴散模型和消除應用

One-dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications

摘要

Support