闭环：面向自回归生成式渲染的无训练回访一致性

摘要

近期条件视频生成模型在将3D引擎渲染（如深度图和无纹理几何体）转化为逼真视频方面展现出巨大潜力，可应用于游戏和沉浸式内容创作。这类应用需要长时序自回归生成，即在保持持久3D世界的同时连续合成新帧。自回归生成器通过有限键值缓存逐块合成视频，因此当摄像机在其上下文被驱逐后重新访问某位置时，即使条件渲染结果（如深度图）与底层几何完美对齐，模型也常会生成不一致的外观。我们通过利用3D引擎已提供的对应关系来解决这种重访不一致性，无需任何后训练：时序对应关系将姿态匹配的历史潜变量块检索至键值缓存中作为闭环记忆；而通过相机姿态和深度重投影获得的空间对应关系，则使令牌级注意力偏向被检索块中几何对应的区域。我们在从TartanAir和TartanGround数据集中挖掘的闭环轨迹上验证了该方法，以模拟复杂的现实应用场景。结果表明，与现有免训练基线相比，该方法在保持整体视频质量的同时，在重访一致性上表现更优。项目页面：https://wenchao-m.github.io/ClosetheLoop.github.io/

English

Recent conditional video generation models have shown promising potentials to transform 3D engine renderings, such as depth maps and untextured geometry, into photorealistic videos for gaming and immersive content creation. These applications require long-horizon auto-regressive generation that continuously synthesizes new frames while preserving a persistent 3D world. Auto-regressive generators synthesize video chunk by chunk with a bounded KV cache, so when the camera revisits a location after its context has been evicted, the model often regenerates inconsistent appearance, even though the conditioning renderings (e.g., depth) remain perfectly aligned with the underlying geometry.We address this revisit inconsistency without any post-training by exploiting correspondences the 3D engine already provides: temporal correspondence retrieves pose-matched historical latent chunks into the KV cache as loop-closure memory, while spatial correspondence from camera pose and depth reprojection biases token-level attention toward geometrically corresponding regions of the retrieved chunks. We demonstrate our method on loop-closure trajectories mined from TartanAir and TartanGround dataset to mirror complicate real-world application scenarios, where it outperforms existing training-free baselines on revisit consistency without losing overall video quality. Project Page: https://wenchao-m.github.io/ClosetheLoop.github.io/