Super Star：面向數字人的流式實時互動智能體

摘要

現有的共語手勢生成方法主要是在離線設定下研究，手勢從完整的語音片段中合成。然而，真實世界場景中的互動式數位人類需要在嚴格的延遲限制下，僅使用當前可用的回應語音，線上生成與語音同步的手勢。因此，先前的方法不適合即時互動，因為它們要么依賴未來的語音資訊，要么產生大量的推論延遲。在本文中，我們針對互動式數位人類定義了線上共語手勢生成問題，並提出了一個即時互動框架，該框架將串流語音回應模組與線上手勢生成模組結合。具體來說，手勢生成器被設計為一個因果多模態自迴歸模型，可從串流回應語音和動作歷史中預測身體動作，從而在無需存取未來語音的情況下實現低延遲且與語音對齊的手勢合成。為了支援此設定，我們進一步提出了一條專為虛擬伴侶場景設計的離線資料合成管線，該管線利用具備主題與情緒感知的主體語料庫來建構多樣化的人機對話，然後根據代理回應生成共語手勢。此外，為了縮短離線資料建構與線上部署之間的差距，我們建立了一個自我演化的訓練迴圈，將線上互動期間收集的使用者回饋納入資料生成過程，從而實現對使用者偏好的持續適應。大量實驗表明，與具有競爭力的現有基線方法相比，我們的框架在延遲與品質的取捨上表現更優、語音與動作同步性更強，並獲得了更高的使用者偏好。專案頁面：https://super-star-2026.github.io/

English

Existing co-speech gesture generation methods are predominantly studied in offline settings, where gestures are synthesized from complete speech segments. However, interactive digital humans in real-world scenarios are required to generate speech-synchronous gestures online, using only currently available response audio under strict latency constraints. As a result, prior methods are unsuitable for real-time interaction, as they either rely on future speech information or incur substantial inference delay. In this paper, we formulate online co-speech gesture generation for interactive digital humans and propose a real-time interactive framework that couples a streaming speech response module with an online gesture generation module. Specifically, the gesture generator is designed as a causal multimodal autoregressive model that predicts body motion from streaming response speech and motion history, enabling low-latency and speech-aligned gesture synthesis without access to future speech. To support this setting, we further propose an offline data synthesis pipeline tailored to virtual companion scenarios, which leverages topic- and emotion-aware subject corpora to construct diverse human-agent dialogues and then generates co-speech gestures conditioned on the agent responses. Moreover, to bridge the gap between offline data construction and online deployment, we establish a self-evolving training loop by incorporating user feedback collected during online interaction into the data generation process, enabling continual adaptation to user preferences. Extensive experiments demonstrate that our framework achieves superior better latency-quality trade-off, stronger speech-motion synchronization, and higher user preference than competitive existing baselines. Project Page: https://super-star-2026.github.io/