InstanceControl: 可控的複雜圖像生成無需實例標註

摘要

可控影像生成方法（如ControlNet）已展现出引入深度图等视觉条件以引导影像生成的显著能力。然而，这些方法在处理包含多实例的复杂场景时常遭遇困境，往往导致实例间的属性混淆。尽管近期研究尝试通过人工实例标注来缓解此问题，但这类标注需求劳动密集。本文提出InstanceControl，一种无需实例标注的新型多实例可控生成方法。我们指出现有方法的主要瓶颈在于无法将实例描述与视觉条件中对应的区域准确关联。为解决此问题，我们借助视觉语言模型（VLM）在文本提示与视觉条件之间建立实例级对应关系。具体而言，VLM自动从文本提示中解析实例描述，同时基于视觉条件预测实例掩模。此外，鉴于预测掩模可能包含噪声，我们引入自适应掩模优化策略，在生成过程中动态精炼这些实例掩模。大量实验表明，我们的方法优于现有最优技术，实现了更高的保真度与精准的实例级控制。

English

Controllable image generation methods, such as ControlNet, have demonstrated a remarkable capacity to introduce visual conditions(e.g., depth maps) to guide image generation. However, these methods often struggle with complex multi-instance scenes, frequently leading to attribute confusion among instances. While recent approaches attempt to mitigate this via manual instance labeling, such requirements are labor-intensive. In this paper, we propose InstanceControl, a novel multi-instance controllable generation method that eliminates the need for instance labeling. We identify the primary bottleneck in existing methods as the inability to accurately associate instance descriptions with their corresponding regions within visual conditions. To address this, we leverage the Vision-Language Model (VLM) to establish instance-level correspondences between text prompts and visual conditions. Specifically, the VLM automatically parses instance descriptions from the text prompts and simultaneously predicts instance masks based on the visual conditions. Furthermore, since the predicted masks may contain noise, we introduce an adaptive mask refinement strategy that dynamically refines these instance masks during the generation process. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods, achieving superior fidelity and precise instance-level control.