基于区域感知双模态直接偏好优化的组合式文本到图像生成

摘要

尽管文本到图像（T2I）模型取得了快速进展，但在生成准确反映复杂组合提示（涵盖属性绑定、对象关系、计数等）的图像方面仍面临挑战。为解决这一问题，我们提出了BiDPO框架，旨在增强T2I模型在组合式文本到图像生成中的能力。我们首先引入了一种精心设计的流水线，用于构建大规模偏好数据集BiComp，并严格进行质量控制。随后，我们将扩散DPO扩展到联合优化图像和文本偏好，实验证明该方法在提升模型遵循复杂文本提示生成图像方面非常有效。为进一步实现细粒度对齐，我们采用了一种区域级引导方法，聚焦与组合概念相关的区域。实验结果表明，我们的BiDPO显著提高了组合保真度，在多个基准测试中均优于以往方法。本方法凸显了基于偏好的微调在复杂文本到图像任务中的潜力，为现有技术提供了一种灵活且可扩展的替代方案。

English

Despite the rapid progress of text-to-image (T2I) models, generating images that accurately reflect complex compositional prompts (covering attribute bindings, object relationships, counting) still remains challenging. To address this, we propose BiDPO, a framework to enhance T2I model's capability of compositional text-to-image generation. We begin by introducing an carefully designed pipeline to construct a large-scale preference dataset, BiComp, with strictly quality control. Then, we extend Diffusion DPO to jointly optimize image and text preferences, which is shown to greatly effective in improving the models to follow complex text prompt in generation. To further enhance the models for fine-grained alignment, we employ a region-level guidance method to focus on regions relevant to compositional concepts. Experimental results demonstrate that our BiDPO substantially improves compositional fidelity, consistently outperforming prior methods across multiple benchmarks. Our approach highlights the potential of preference-based fine-tuning for complex text-to-image tasks, offering a flexible and scalable alternative to existing techniques.