故事適配器：一個無需訓練的長故事可視化迭代框架

Story-Adapter: A Training-free Iterative Framework for Long Story Visualization

October 8, 2024

作者: Jiawei Mao, Xiaoke Huang, Yunfei Xie, Yuanqi Chang, Mude Hui, Bingjie Xu, Yuyin Zhou

cs.AI

摘要

故事視覺化是根據敘述生成連貫圖像的任務，在文本轉圖像模型，尤其是擴散模型的出現下取得了顯著進展。然而，在長篇故事視覺化（即長達100幀）中，保持語義一致性、生成高質量細緻交互作用以及確保計算可行性仍然具有挑戰性。在這項工作中，我們提出了一個無需訓練且計算效率高的框架，稱為Story-Adapter，以增強長篇故事的生成能力。具體而言，我們提出了一種迭代範式來改進每個生成的圖像，利用文本提示和前一次迭代中生成的所有圖像。我們框架的核心是一個無需訓練的全局參考交叉注意力模塊，它匯總了前一次迭代中生成的所有圖像，以保持整個故事的語義一致性，同時通過全局嵌入來降低計算成本。這種迭代過程通過反覆納入文本約束逐步優化圖像生成，從而產生更精確和細緻的交互作用。大量實驗驗證了Story-Adapter在提高語義一致性和生成能力，尤其是在長篇故事情境中改善細緻交互作用方面的優越性。項目頁面和相關代碼可通過 https://jwmao1.github.io/storyadapter 訪問。

English

Story visualization, the task of generating coherent images based on a narrative, has seen significant advancements with the emergence of text-to-image models, particularly diffusion models. However, maintaining semantic consistency, generating high-quality fine-grained interactions, and ensuring computational feasibility remain challenging, especially in long story visualization (i.e., up to 100 frames). In this work, we propose a training-free and computationally efficient framework, termed Story-Adapter, to enhance the generative capability of long stories. Specifically, we propose an iterative paradigm to refine each generated image, leveraging both the text prompt and all generated images from the previous iteration. Central to our framework is a training-free global reference cross-attention module, which aggregates all generated images from the previous iteration to preserve semantic consistency across the entire story, while minimizing computational costs with global embeddings. This iterative process progressively optimizes image generation by repeatedly incorporating text constraints, resulting in more precise and fine-grained interactions. Extensive experiments validate the superiority of Story-Adapter in improving both semantic consistency and generative capability for fine-grained interactions, particularly in long story scenarios. The project page and associated code can be accessed via https://jwmao1.github.io/storyadapter .