基於區域感知雙模態直接偏好優化的組合式文字到圖像生成

摘要

儘管文字到圖像（T2I）模型進展迅速，但在生成準確反映複雜組合提示（涵蓋屬性綁定、物體關係、計數）的圖像方面仍具挑戰。為解決此問題，我們提出 BiDPO 框架，旨在增強 T2I 模型的複合式文字到圖像生成能力。首先，我們引入精心設計的流程，建構大規模偏好資料集 BiComp，並嚴格控管品質。接著，我們擴展擴散 DPO（Diffusion DPO）以聯合最佳化圖像與文字偏好，實驗證明此方法能有效提升模型遵循複雜文字提示進行生成的能力。為進一步強化模型在細粒度對齊上的表現，我們採用區域級引導方法，聚焦於與複合概念相關的區域。實驗結果顯示，我們的 BiDPO 顯著提升複合忠實度，在多項基準測試中持續優於既有方法。本方法凸顯了基於偏好的微調在複雜文字到圖像任務中的潛力，為現有技術提供了靈活且可擴展的替代方案。

English

Despite the rapid progress of text-to-image (T2I) models, generating images that accurately reflect complex compositional prompts (covering attribute bindings, object relationships, counting) still remains challenging. To address this, we propose BiDPO, a framework to enhance T2I model's capability of compositional text-to-image generation. We begin by introducing an carefully designed pipeline to construct a large-scale preference dataset, BiComp, with strictly quality control. Then, we extend Diffusion DPO to jointly optimize image and text preferences, which is shown to greatly effective in improving the models to follow complex text prompt in generation. To further enhance the models for fine-grained alignment, we employ a region-level guidance method to focus on regions relevant to compositional concepts. Experimental results demonstrate that our BiDPO substantially improves compositional fidelity, consistently outperforming prior methods across multiple benchmarks. Our approach highlights the potential of preference-based fine-tuning for complex text-to-image tasks, offering a flexible and scalable alternative to existing techniques.