HelloWorld：在視頻世界模型中實現社交互動角色

摘要

儘管近期影片世界模型取得了顯著進展，但使用者與這些世界中的角色之間的社交互動仍未被支援。為填補此缺口，我們提出 HelloWorld，一個能與世界內角色進行社交互動的影片世界模型。只需按一下按鈕，使用者即可提示畫面上的角色朝向鏡頭回應，例如轉向觀看者、揮手、點頭，或說出簡短的問候語。為了讓這些互動更自然，我們提出一個自我蒸餾（self-distillation）流程，在模型自身合成的資料上微調影片生成模型。每個合成片段同時包含社交互動與鏡頭運動，使模型能學習鏡頭姿態條件（camera-pose conditioning），而不會降低互動品質。在推論時，我們進一步引入一個免訓練（training-free）模組，用於判斷互動發生的時間點。一旦按下按鈕，該模組會調節 DiT 的交叉注意力遮罩（cross-attention masks），使與互動相關的文字提示僅關注按壓時間窗內的幀，從時間上定位角色的回應。我們亦建立 HelloWorldBench，一個包含 400 筆樣本的基準測試，具備三項社交互動指標與三項傳統指標，以供評估。實驗證明，HelloWorld 在互動品質上超越多種基線方法，同時維持最先進的畫面美學與鏡頭姿態跟隨能力。專案頁面：https://github.com/AlayaLab/HelloWorld

English

Despite the remarkable recent progress of video world models, social interaction between users and the characters within these worlds remains unsupported. To fill this gap, we present HelloWorld, a video world model that enables social interaction with in-world characters. With a single button press, users can prompt the on-screen character to respond toward the camera, e.g., turning to the viewer, waving, nodding, or speaking a short greeting. To make these interactions natural, we propose a self-distillation pipeline that finetunes the video generation model on data synthesized by itself. Each synthesized clip contains both social interactions and camera motion, allowing the model to learn camera-pose conditioning without degrading interaction quality. At inference, we further introduce a training-free module that determines when the interaction occurs. Upon a button press, it modulates the cross-attention masks of the DiT so that the interaction-related text prompt attends only to the frames within the press window, temporally localizing the character's response. We further build HelloWorldBench, a 400-sample benchmark with three social interaction metrics alongside three conventional metrics, for evaluation. Experiments demonstrate that HelloWorld surpasses a variety of baselines in interaction quality, while maintaining state-of-the-art picture aesthetics and camera-pose following. Project page: https://github.com/AlayaLab/HelloWorld