UnityShots：記憶驅動的多鏡頭音視頻生成及邊界感知門控

摘要

要生成连贯的多鏡頭影片，需要結構化的跨鏡頭記憶。主體外觀、場景脈絡與說話者身份必須在鏡頭切換間保持一致。現有方法若非以固定長度序列進行端到端訓練而無法擴展，就是透過線性增長的記憶庫逐鏡頭生成，或是利用LLM規劃器協調預訓練生成器卻缺乏感知多鏡頭的骨幹網路。我們提出UnityShots，這套基於LTX-2.3打造的記憶驅動多鏡頭音像生成系統，並使用標註過的電影與音樂錄影帶鏡頭進行訓練。影像流維持兩個固定大小的記憶槽：長期記憶槽錨定於開場鏡頭，短期記憶槽則保存緊鄰的前一個結尾；每次鏡頭切換時，由邊界條件門控（融合影像切換機率與節拍追蹤訊號）更新兩者。音訊流則在每個鏡頭注入參考說話者權杖，以保留音色特徵，無需滑動音訊庫。透過AdaLN學習到的離散切換類型先驗，可在推論時作為控制鏡頭過渡強度的旋鈕。我們發布一套基準測試，包含200組跨文化多鏡頭序列，涵蓋六大族群區域與十種以上語言，並提供每鏡頭的參考身份、參考音訊，以及每個邊界的過渡標籤。在I2V、T2V與R2V三種條件模式下評估，UnityShots在所有跨鏡頭連貫性指標上領先開源基準，並在多鏡頭面向比肩最強的閉源系統。

English

Generating a coherent multi-shot video requires structured cross-shot memory. Subject appearance, scene context, and speaker identity must persist across cuts. Existing approaches either train end-to-end over fixed-length sequences and cannot scale, generate shot-by-shot with memory banks that grow linearly, or orchestrate pretrained generators under an LLM planner without a multi-shot-aware backbone. We present UnityShots, a memory-driven multi-shot audio-video generation system built on LTX-2.3, trained on annotated cinematic and music-video shots. The video stream maintains two fixed-size slots, a long-term memory (LTM) slot anchored to the opening shot and a short-term memory (STM) slot holding the immediately preceding tail, both updated at every cut by a boundary-conditioned gate that fuses visual cut probability and beat-tracker signals. The audio stream injects a reference speaker token at every shot to preserve vocal timbre without a sliding audio bank. A discrete cut-type prior, learned through AdaLN, becomes an inference-time control knob over transition strength. We release a benchmark of 200 multi-cultural multi-shot sequences spanning six ethnic regions and ten or more languages, with per-shot reference identities, reference audio, and per-boundary transition labels. Evaluated across I2V, T2V, and R2V conditioning modes, UnityShots leads open-source baselines on every cross-shot coherence metric and matches the strongest closed-source system on the multi-shot axes.