一维适配器：概念、扩散模型和擦除应用

摘要

商业和开源扩散模型（DMs）在文本到图像生成中的普遍使用促使风险缓解，以防止不良行为。学术界现有的概念擦除方法都基于完全参数或基于规范的微调，我们观察到以下问题：1）生成向侵蚀的变化：目标消除期间的参数漂移导致变化和潜在变形在所有生成中，甚至以不同程度侵蚀其他概念，这在多概念擦除时更为明显；2）转移能力不足和部署效率低：先前的特定于模型的擦除阻碍了概念的灵活组合和向其他模型的无需训练的转移，导致线性成本随着部署场景的增加而增长。为实现非侵入性、精确、可定制和可转移的消除，我们将我们的擦除框架基于一维适配器，一次从大多数DMs中擦除多个概念，适用于各种擦除应用。概念半透膜结构被注入为膜（SPM）到任何DM中学习有针对性的擦除，同时通过一种新颖的潜在锚定微调策略有效地缓解了变化和侵蚀现象。一旦获得，SPMs可以灵活组合并即插即用于其他DMs，无需特定的重新调整，实现及时和高效地适应各种情景。在生成过程中，我们的促进传输机制动态调节每个SPM的渗透性以响应不同的输入提示，进一步减少对其他概念的影响。在大约40个概念、7个DMs和4个擦除应用中的定量和定性结果已经证明了SPM的卓越擦除效果。我们的代码和预调整的SPMs将在项目页面https://lyumengyao.github.io/projects/spm 上提供。

English

The prevalent use of commercial and open-source diffusion models (DMs) for text-to-image generation prompts risk mitigation to prevent undesired behaviors. Existing concept erasing methods in academia are all based on full parameter or specification-based fine-tuning, from which we observe the following issues: 1) Generation alternation towards erosion: Parameter drift during target elimination causes alternations and potential deformations across all generations, even eroding other concepts at varying degrees, which is more evident with multi-concept erased; 2) Transfer inability & deployment inefficiency: Previous model-specific erasure impedes the flexible combination of concepts and the training-free transfer towards other models, resulting in linear cost growth as the deployment scenarios increase. To achieve non-invasive, precise, customizable, and transferable elimination, we ground our erasing framework on one-dimensional adapters to erase multiple concepts from most DMs at once across versatile erasing applications. The concept-SemiPermeable structure is injected as a Membrane (SPM) into any DM to learn targeted erasing, and meantime the alteration and erosion phenomenon is effectively mitigated via a novel Latent Anchoring fine-tuning strategy. Once obtained, SPMs can be flexibly combined and plug-and-play for other DMs without specific re-tuning, enabling timely and efficient adaptation to diverse scenarios. During generation, our Facilitated Transport mechanism dynamically regulates the permeability of each SPM to respond to different input prompts, further minimizing the impact on other concepts. Quantitative and qualitative results across ~40 concepts, 7 DMs and 4 erasing applications have demonstrated the superior erasing of SPM. Our code and pre-tuned SPMs will be available on the project page https://lyumengyao.github.io/projects/spm.

一维适配器：概念、扩散模型和擦除应用

One-dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications

摘要

Support