閉環：無訓練回顧一致性於自回歸生成渲染

摘要

近期基於條件的視頻生成模型已展現出將三維引擎渲染（如深度圖與未紋理幾何）轉換為逼真影片的巨大潛力，可應用於遊戲與沉浸式內容創作。此類應用需要具備長程自回歸生成能力，能在持續合成新畫面的同時，維持穩定的三維世界表徵。自回歸生成器透過有限大小的鍵值緩存逐塊合成視頻，因此當相機在環境上下文被清除後重新訪問某個位置時，即使條件渲染（如深度）仍與底層幾何完美對齊，模型仍常生成不一致的外觀。我們提出一種無需後訓練的方法，利用三維引擎已提供的對應關係來解決此重訪不一致性：時間對應關係可將姿態匹配的歷史潛變量塊檢索至鍵值緩存中，作為回環閉合記憶；而源自相機姿態與深度重新投影的空間對應關係，則能引導Token級注意力偏向檢索塊中幾何對應的區域。我們在從TartanAir與TartanGround數據集中提取的回環閉合軌跡上驗證方法，以模擬複雜的真實應用場景。結果顯示，本方法在不犧牲整體視頻品質的前提下，在重訪一致性上優於現有無訓練基線。項目頁面：https://wenchao-m.github.io/ClosetheLoop.github.io/

English

Recent conditional video generation models have shown promising potentials to transform 3D engine renderings, such as depth maps and untextured geometry, into photorealistic videos for gaming and immersive content creation. These applications require long-horizon auto-regressive generation that continuously synthesizes new frames while preserving a persistent 3D world. Auto-regressive generators synthesize video chunk by chunk with a bounded KV cache, so when the camera revisits a location after its context has been evicted, the model often regenerates inconsistent appearance, even though the conditioning renderings (e.g., depth) remain perfectly aligned with the underlying geometry.We address this revisit inconsistency without any post-training by exploiting correspondences the 3D engine already provides: temporal correspondence retrieves pose-matched historical latent chunks into the KV cache as loop-closure memory, while spatial correspondence from camera pose and depth reprojection biases token-level attention toward geometrically corresponding regions of the retrieved chunks. We demonstrate our method on loop-closure trajectories mined from TartanAir and TartanGround dataset to mirror complicate real-world application scenarios, where it outperforms existing training-free baselines on revisit consistency without losing overall video quality. Project Page: https://wenchao-m.github.io/ClosetheLoop.github.io/