IPAdapter-Instruct：使用Instruct提示解决基于图像条件的歧义

IPAdapter-Instruct: Resolving Ambiguity in Image-based Conditioning using Instruct Prompts

August 6, 2024

作者: Ciara Rowles, Shimon Vainer, Dante De Nigris, Slava Elizarov, Konstantin Kutsy, Simon Donné

cs.AI

摘要

扩散模型不断推动最先进的图像生成边界，但这个过程很难以任何细微之处进行控制：实践证明，文本提示无法准确描述图像风格或细微的结构细节（如面部）。ControlNet和IPAdapter解决了这一不足，通过在生成过程中对图像进行条件设定，但每个单独实例仅限于对单个条件后验进行建模：对于需要在同一工作流程中实现多个不同后验的实际用例，训练和使用多个适配器很繁琐。我们提出了IPAdapter-Instruct，它将自然图像条件设定与“指令”提示相结合，以在相同条件图像之间切换解释：风格转移、对象提取、两者，或者其他什么？IPAdapter-Instruct能够高效地学习多个任务，与专门的每个任务模型相比，几乎不会损失质量。

English

Diffusion models continuously push the boundary of state-of-the-art image generation, but the process is hard to control with any nuance: practice proves that textual prompts are inadequate for accurately describing image style or fine structural details (such as faces). ControlNet and IPAdapter address this shortcoming by conditioning the generative process on imagery instead, but each individual instance is limited to modeling a single conditional posterior: for practical use-cases, where multiple different posteriors are desired within the same workflow, training and using multiple adapters is cumbersome. We propose IPAdapter-Instruct, which combines natural-image conditioning with ``Instruct'' prompts to swap between interpretations for the same conditioning image: style transfer, object extraction, both, or something else still? IPAdapterInstruct efficiently learns multiple tasks with minimal loss in quality compared to dedicated per-task models.