PlayerOne：第一人称世界模拟器

摘要

我们推出PlayerOne，首个以自我为中心的逼真世界模拟器，它能够在生动动态的环境中实现沉浸式且无限制的探索。基于用户提供的自我中心场景图像，PlayerOne能够精确构建对应世界，并生成与外部摄像机捕捉到的用户真实场景人体运动严格对齐的自我中心视角视频。PlayerOne采用由粗到精的训练流程，首先在大规模自我中心文本-视频对上进行预训练，以获得粗粒度的自我中心理解，随后利用我们自动构建的流程，在从自我-外部中心视频数据集中提取的同步运动-视频数据上进行微调。此外，考虑到不同组件的重要性差异，我们设计了一种部分解耦的运动注入方案，实现了对局部运动的精确控制。同时，我们开发了一个联合重建框架，逐步建模4D场景与视频帧，确保长视频生成中的场景一致性。实验结果展示了其在精确控制多样化人体运动及世界一致性建模多种场景方面的卓越泛化能力。这一成果标志着自我中心真实世界模拟的首次尝试，为社区探索世界建模及其多样化应用的新领域铺平了道路。

English

We introduce PlayerOne, the first egocentric realistic world simulator, facilitating immersive and unrestricted exploration within vividly dynamic environments. Given an egocentric scene image from the user, PlayerOne can accurately construct the corresponding world and generate egocentric videos that are strictly aligned with the real scene human motion of the user captured by an exocentric camera. PlayerOne is trained in a coarse-to-fine pipeline that first performs pretraining on large-scale egocentric text-video pairs for coarse-level egocentric understanding, followed by finetuning on synchronous motion-video data extracted from egocentric-exocentric video datasets with our automatic construction pipeline. Besides, considering the varying importance of different components, we design a part-disentangled motion injection scheme, enabling precise control of part-level movements. In addition, we devise a joint reconstruction framework that progressively models both the 4D scene and video frames, ensuring scene consistency in the long-form video generation. Experimental results demonstrate its great generalization ability in precise control of varying human movements and worldconsistent modeling of diverse scenarios. It marks the first endeavor into egocentric real-world simulation and can pave the way for the community to delve into fresh frontiers of world modeling and its diverse applications.

PlayerOne：第一人称世界模拟器

PlayerOne: Egocentric World Simulator

摘要

Support