HelloWorld：在视频世界模型中实现社交互动角色

摘要

尽管视频世界模型近期取得了显著进展，但用户与这些世界中的角色之间的社交互动仍未得到支持。为填补这一空白，我们提出了HelloWorld，一种支持与世界中角色进行社交互动的视频世界模型。用户只需按一下按钮，即可提示屏幕中的角色面向摄像头做出回应，例如转向观众、挥手、点头或说一句简短的问候。为了使这些互动自然流畅，我们提出了一种自蒸馏流水线，在模型自身合成的数据上对视频生成模型进行微调。每个合成片段同时包含社交互动和相机运动，使模型能够学习相机姿态条件控制，而不会降低互动质量。在推理阶段，我们进一步引入了一个无需训练模块，用于确定互动发生的时间。当按下按钮时，该模块调制DiT的交叉注意力掩码，使与互动相关的文本提示仅关注按下窗口内的帧，从而在时间上定位角色的回应。我们进一步构建了HelloWorldBench，一个包含400个样本的基准测试，包含三个社交互动指标和三个常规指标，用于评估。实验表明，HelloWorld在互动质量上超越了多种基线方法，同时保持了最先进的画面美学和相机姿态跟随能力。项目页面：https://github.com/AlayaLab/HelloWorld

English

Despite the remarkable recent progress of video world models, social interaction between users and the characters within these worlds remains unsupported. To fill this gap, we present HelloWorld, a video world model that enables social interaction with in-world characters. With a single button press, users can prompt the on-screen character to respond toward the camera, e.g., turning to the viewer, waving, nodding, or speaking a short greeting. To make these interactions natural, we propose a self-distillation pipeline that finetunes the video generation model on data synthesized by itself. Each synthesized clip contains both social interactions and camera motion, allowing the model to learn camera-pose conditioning without degrading interaction quality. At inference, we further introduce a training-free module that determines when the interaction occurs. Upon a button press, it modulates the cross-attention masks of the DiT so that the interaction-related text prompt attends only to the frames within the press window, temporally localizing the character's response. We further build HelloWorldBench, a 400-sample benchmark with three social interaction metrics alongside three conventional metrics, for evaluation. Experiments demonstrate that HelloWorld surpasses a variety of baselines in interaction quality, while maintaining state-of-the-art picture aesthetics and camera-pose following. Project page: https://github.com/AlayaLab/HelloWorld