PanoWorld：真實世界全景生成

摘要

在本研究中，我們旨在透過利用全向表示中的旋轉等變性質來解決全景世界模型中的長程記憶挑戰，其中旋轉可視為一種隱含的幾何變換。基於此洞察，我們提出PanoWorld，透過固定航向將相機軌跡簡化為平移，並藉由密集全景光線條件（DPRC）與幾何感知記憶增強（GMA）實現當前動作建模及長程記憶。接著，引入三階段訓練流程逐步優化各組件。為在現有數據集相對穩定的情況下，更好評估大規模空間變化與多樣光照條件下的物理一致性，我們建構World360——一個大規模數據集，包含透過全景無人機收集的真實世界影片片段，以及由AirSim360生成的高品質模擬片段。在World360上的大量實驗證明了PanoWorld的有效性，其表現大幅優於其他替代方法。我們的模型、訓練程式碼及數據集將對外公開。更多資訊可參閱專案頁面：https://lihaoy-ux.github.io/panoworld-page/。

English

In this work, we aim to address the challenge of long-range memory in panoramic world models by exploiting the rotation-equivariant property of omnidirectional representations, where rotation can be treated as an implicit geometric transformation.Building on this insight, we propose PanoWorld, which simplifies camera trajectories into translations via fixed headings for both current-action modeling and long-range memory through Dense Panoramic Ray-Conditioning (DPRC) and Geometry-aware Memory Augmentation (GMA).Then, a three-stage training pipeline is introduced to progressively optimize each component. To better evaluate physical consistency under large-scale spatial variations and diverse illumination conditions, where existing datasets are relatively stable, we construct World360, a large-scale dataset consisting of both real-world video clips collected via panoramic unmanned aerial vehicles and high-quality simulated clips generated by AirSim360.Extensive experiments on World360 demonstrate the effectiveness of PanoWorld, outperforming alternative methods by a large margin.Our models, training code, and dataset will be publicly available. More information can be found on our project page: https://lihaoy-ux.github.io/panoworld-page/.