一维适配器统一架构：概念、扩散模型及其擦除应用

摘要

当前商业及开源扩散模型在文本到图像生成领域的广泛应用，促使人们需要采取风险规避措施以防止不良生成行为。学术界现有的概念擦除方法均基于全参数或特定规范微调，我们从中观察到以下问题：1）生成结果趋向异化：目标消除过程中的参数漂移会导致所有生成内容发生改变甚至畸变，不同程度地侵蚀其他概念，这种现象在多概念擦除时更为明显；2）迁移障碍与部署低效：既往模型特定的擦除方式阻碍了概念的灵活组合及向其他模型的免训练迁移，导致部署场景增加时成本呈线性增长。为实现非侵入式、精准可定制且可迁移的概念消除，我们基于一维适配器构建擦除框架，可在多样化擦除应用中一次性从多数扩散模型中消除多个概念。通过将概念半透膜结构作为膜层注入任意扩散模型，在实现目标概念擦除的同时，采用新颖的潜在锚定微调策略有效缓解生成异化与概念侵蚀现象。所得半透膜既可灵活组合，也能即插即用地迁移至其他扩散模型而无需专门重新调参，从而实现对多样化场景的即时高效适配。在生成过程中，我们的促进传输机制会动态调节各半透膜的渗透性以响应不同输入提示，进一步降低对其他概念的影响。在约40个概念、7种扩散模型和4类擦除应用上的定量与定性实验表明，半透膜结构具有卓越的擦除性能。我们的代码与预训练半透膜将在项目页面https://lyumengyao.github.io/projects/spm开放获取。

English

The prevalent use of commercial and open-source diffusion models (DMs) for text-to-image generation prompts risk mitigation to prevent undesired behaviors. Existing concept erasing methods in academia are all based on full parameter or specification-based fine-tuning, from which we observe the following issues: 1) Generation alternation towards erosion: Parameter drift during target elimination causes alternations and potential deformations across all generations, even eroding other concepts at varying degrees, which is more evident with multi-concept erased; 2) Transfer inability & deployment inefficiency: Previous model-specific erasure impedes the flexible combination of concepts and the training-free transfer towards other models, resulting in linear cost growth as the deployment scenarios increase. To achieve non-invasive, precise, customizable, and transferable elimination, we ground our erasing framework on one-dimensional adapters to erase multiple concepts from most DMs at once across versatile erasing applications. The concept-SemiPermeable structure is injected as a Membrane (SPM) into any DM to learn targeted erasing, and meantime the alteration and erosion phenomenon is effectively mitigated via a novel Latent Anchoring fine-tuning strategy. Once obtained, SPMs can be flexibly combined and plug-and-play for other DMs without specific re-tuning, enabling timely and efficient adaptation to diverse scenarios. During generation, our Facilitated Transport mechanism dynamically regulates the permeability of each SPM to respond to different input prompts, further minimizing the impact on other concepts. Quantitative and qualitative results across ~40 concepts, 7 DMs and 4 erasing applications have demonstrated the superior erasing of SPM. Our code and pre-tuned SPMs will be available on the project page https://lyumengyao.github.io/projects/spm.

一维适配器统一架构：概念、扩散模型及其擦除应用

One-dimensional Adapter to Rule Them All: Concepts, Diffusion Models and Erasing Applications

摘要

Support