理解与生成：多模态模型中的优化困境探析

摘要

当前多模态模型研究面临一个关键挑战：提升生成能力往往以牺牲理解为代价，反之亦然。我们通过分析发现，这一权衡关系的根本原因可能在于生成与理解之间的潜在冲突，这种冲突在模型内部形成了竞争动态。为此，我们提出"推理-反思-优化"（R3）框架。该创新算法将单步生成任务重构为"生成-理解-再生成"的多步过程，通过显式利用模型在生成过程中的理解能力，成功缓解了优化困境，不仅获得了更强的生成效果，还提升了与生成过程相关的理解能力。这一研究为设计新一代统一多模态模型提供了重要启示。代码已开源：https://github.com/sen-ye/R3。

English

Current research in multimodal models faces a key challenge where enhancing generative capabilities often comes at the expense of understanding, and vice versa. We analyzed this trade-off and identify the primary cause might be the potential conflict between generation and understanding, which creates a competitive dynamic within the model. To address this, we propose the Reason-Reflect-Refine (R3) framework. This innovative algorithm re-frames the single-step generation task into a multi-step process of "generate-understand-regenerate". By explicitly leveraging the model's understanding capability during generation, we successfully mitigate the optimization dilemma, achieved stronger generation results and improved understanding ability which are related to the generation process. This offers valuable insights for designing next-generation unified multimodal models. Code is available at https://github.com/sen-ye/R3.

理解与生成：多模态模型中的优化困境探析

Understanding vs. Generation: Navigating Optimization Dilemma in Multimodal Models

摘要

Support