一維適配器統御萬法：概念、擴散模型與抹除應用

摘要

當前商業及開源擴散模型在文本到圖像生成領域的廣泛應用，促使我們需要採取風險緩解措施來防止不良行為。學術界現有的概念抹除方法均基於全參數或特定規格的微調，我們從中觀察到以下問題：1）生成結果的侵蝕性異變：目標消除過程中的參數漂移會導致所有生成內容發生變異甚至畸變，並不同程度地侵蝕其他概念，這一現象在多概念抹除時更為明顯；2）遷移失效與部署低效：既往的模型專用抹除方式阻礙了概念的靈活組合與免訓練跨模型遷移，導致部署場景增加時成本呈線性增長。為實現非侵入式、精準可定製且可遷移的抹除效果，我們基於一維適配器構建抹除框架，能夠一次性從多數擴散模型中抹除多個概念，適用於各類抹除應用場景。通過將概念半透性結構作為薄膜注入任意擴散模型，使其學習目標抹除任務，同時採用新穎的潛空間錨定微調策略有效緩解異變與侵蝕現象。一旦訓練完成，SPM薄膜可靈活組合並即插即用於其他擴散模型，無需針對性重新調優，從而實現對多樣化場景的及時高效適配。在生成過程中，我們的促進傳輸機制會動態調控每個SPM薄膜的滲透性以響應不同輸入提示，進一步最小化對其他概念的影響。在約40個概念、7種擴散模型及4類抹除應用中進行的定量與定性實驗，均證實了SPM薄膜的卓越抹除能力。我們的代碼與預訓練SPM薄膜將在項目頁面https://lyumengyao.github.io/projects/spm公開。

English

The prevalent use of commercial and open-source diffusion models (DMs) for text-to-image generation prompts risk mitigation to prevent undesired behaviors. Existing concept erasing methods in academia are all based on full parameter or specification-based fine-tuning, from which we observe the following issues: 1) Generation alternation towards erosion: Parameter drift during target elimination causes alternations and potential deformations across all generations, even eroding other concepts at varying degrees, which is more evident with multi-concept erased; 2) Transfer inability & deployment inefficiency: Previous model-specific erasure impedes the flexible combination of concepts and the training-free transfer towards other models, resulting in linear cost growth as the deployment scenarios increase. To achieve non-invasive, precise, customizable, and transferable elimination, we ground our erasing framework on one-dimensional adapters to erase multiple concepts from most DMs at once across versatile erasing applications. The concept-SemiPermeable structure is injected as a Membrane (SPM) into any DM to learn targeted erasing, and meantime the alteration and erosion phenomenon is effectively mitigated via a novel Latent Anchoring fine-tuning strategy. Once obtained, SPMs can be flexibly combined and plug-and-play for other DMs without specific re-tuning, enabling timely and efficient adaptation to diverse scenarios. During generation, our Facilitated Transport mechanism dynamically regulates the permeability of each SPM to respond to different input prompts, further minimizing the impact on other concepts. Quantitative and qualitative results across ~40 concepts, 7 DMs and 4 erasing applications have demonstrated the superior erasing of SPM. Our code and pre-tuned SPMs will be available on the project page https://lyumengyao.github.io/projects/spm.

一維適配器統御萬法：概念、擴散模型與抹除應用

One-dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications

摘要

Support