T-LoRA：无需过拟合的单图像扩散模型定制

摘要

尽管扩散模型微调为定制预训练模型以生成特定对象提供了强大手段，但在训练样本有限时，常出现过拟合问题，这不仅削弱了模型的泛化能力，还影响了输出多样性。本文聚焦于最具挑战性且影响深远的任务——仅用单张概念图像来适配扩散模型，因为单图定制在实际应用中潜力最大。我们提出了T-LoRA，一种时间步依赖的低秩适应框架，专为扩散模型个性化设计。研究表明，较高扩散时间步比低时间步更易过拟合，因此需要一种对时间步敏感的微调策略。T-LoRA包含两大创新：(1) 一种动态微调策略，根据扩散时间步调整秩约束更新；(2) 一种权重参数化技术，通过正交初始化确保适配器组件间的独立性。大量实验证明，T-LoRA及其各组件均优于标准LoRA及其他扩散模型个性化技术，在概念保真度与文本对齐间实现了更优平衡，凸显了T-LoRA在数据有限和资源受限场景下的潜力。代码已发布于https://github.com/ControlGenAI/T-LoRA。

English

While diffusion model fine-tuning offers a powerful approach for customizing pre-trained models to generate specific objects, it frequently suffers from overfitting when training samples are limited, compromising both generalization capability and output diversity. This paper tackles the challenging yet most impactful task of adapting a diffusion model using just a single concept image, as single-image customization holds the greatest practical potential. We introduce T-LoRA, a Timestep-Dependent Low-Rank Adaptation framework specifically designed for diffusion model personalization. In our work we show that higher diffusion timesteps are more prone to overfitting than lower ones, necessitating a timestep-sensitive fine-tuning strategy. T-LoRA incorporates two key innovations: (1) a dynamic fine-tuning strategy that adjusts rank-constrained updates based on diffusion timesteps, and (2) a weight parametrization technique that ensures independence between adapter components through orthogonal initialization. Extensive experiments show that T-LoRA and its individual components outperform standard LoRA and other diffusion model personalization techniques. They achieve a superior balance between concept fidelity and text alignment, highlighting the potential of T-LoRA in data-limited and resource-constrained scenarios. Code is available at https://github.com/ControlGenAI/T-LoRA.

T-LoRA：无需过拟合的单图像扩散模型定制

T-LoRA: Single Image Diffusion Model Customization Without Overfitting

摘要

Support