PlayerOne：以自我為中心的世界模擬器

摘要

我們推出了PlayerOne，首個以自我為中心的真實世界模擬器，它能夠在生動動態的環境中促進沉浸式且無限制的探索。基於用戶提供的自我中心場景圖像，PlayerOne能夠精確構建相應的世界，並生成與由外置攝像頭捕捉的用戶真實場景人體運動嚴格對齊的自我中心視頻。PlayerOne採用從粗到細的訓練流程，首先在大規模自我中心文本-視頻對上進行預訓練，以實現粗粒度的自我中心理解，隨後通過我們的自動構建管道從自我-外部中心視頻數據集中提取同步運動-視頻數據進行微調。此外，考慮到不同組件的重要性差異，我們設計了一種部分解耦的運動注入方案，實現了對局部運動的精確控制。同時，我們開發了一個聯合重建框架，逐步建模4D場景和視頻幀，確保在長視頻生成中的場景一致性。實驗結果展示了其在精確控制多樣人體運動和對多種場景進行世界一致性建模方面的強大泛化能力。這標誌著自我中心真實世界模擬的首次嘗試，並為學術界探索世界建模及其多樣應用的新前沿鋪平了道路。

English

We introduce PlayerOne, the first egocentric realistic world simulator, facilitating immersive and unrestricted exploration within vividly dynamic environments. Given an egocentric scene image from the user, PlayerOne can accurately construct the corresponding world and generate egocentric videos that are strictly aligned with the real scene human motion of the user captured by an exocentric camera. PlayerOne is trained in a coarse-to-fine pipeline that first performs pretraining on large-scale egocentric text-video pairs for coarse-level egocentric understanding, followed by finetuning on synchronous motion-video data extracted from egocentric-exocentric video datasets with our automatic construction pipeline. Besides, considering the varying importance of different components, we design a part-disentangled motion injection scheme, enabling precise control of part-level movements. In addition, we devise a joint reconstruction framework that progressively models both the 4D scene and video frames, ensuring scene consistency in the long-form video generation. Experimental results demonstrate its great generalization ability in precise control of varying human movements and worldconsistent modeling of diverse scenarios. It marks the first endeavor into egocentric real-world simulation and can pave the way for the community to delve into fresh frontiers of world modeling and its diverse applications.

PlayerOne：以自我為中心的世界模擬器

PlayerOne: Egocentric World Simulator

摘要

Support