全方位互動代理人技術報告

摘要

在本研究中，我們提出 Gander，一個端到端模型，於單一框架內統一全感知、即時互動與代理能力。相較於回合制的傳統範式，Gander 持續接收跨多模態的串流輸入，包括視訊、語音與文字，從而在日常對話與複雜工作流程導向的代理場景中實現自然的全雙工互動。使用者可隨時打斷模型，模型亦能主動提供中間回饋或提出追問。為原生支援這些能力，Gander 採用兩項關鍵架構設計：1) 其採用小腦-大腦協作框架，其中小腦負責即時互動與全模態對話能力，而大腦負責複雜推理與更高階的代理任務。這兩個組件透過工具呼叫與代理編排執行環境持續互動。2) 小腦建立於串流式 Thinker-Talker 架構之上，使用者輸入與模型輸出進一步在區塊層級被攤平為有序的詞元串流，為低延遲、持續互動提供統一表徵。我們從四個面向對 Gander 進行全面評估：對話能力、全模態理解、互動能力與代理智慧。內部人工評估顯示，Gander 維持 SOTA 開源模型自然且具表現力的口語對話能力，同時在全模態互動方面達到具競爭力的表現。Gander 亦在具挑戰性的真實世界場景中展現穩健性，包括背景噪音干擾、多方互動與附和式溝通。我們釋出 Gander 及其模型、程式碼與資料，以促進社群進一步的研究與開發。

English

In this work, we present Gander, an end-to-end model that unifies omni perception, realtime interaction, and agentic capabilities within a single framework. In contrast to turn-based conventional paradigms, Gander continuously receives streaming inputs across multiple modalities, including video, speech, and text, enabling natural full-duplex interaction in both everyday conversations and complex workflow-oriented agent scenarios. Users can interrupt the model at any time, while the model can also proactively provide intermediate feedback or ask follow up questions. To natively support these capabilities, Gander adopts two key architectural designs: 1) It employs a Cerebellum-Brain collaborative framework, in which the Cerebellum is responsible for realtime interaction and omni conversational capabilities, while the Brain handles complex reasoning and higher-level agentic tasks. The two components interact continuously through tool calling and the agent orchestration runtime. 2) The Cerebellum is built upon a streaming Thinker-Talker architecture, user inputs and model outputs are further flattened into an ordered token stream at the chunk level, providing a unified representation for low latency, continuous interaction. We conduct comprehensive evaluations of Gander across four dimensions: conversational ability, omni understanding, interactive capability, and agentic intelligence. Internal human evaluations demonstrate that Gander maintains the natural and expressive spoken dialogue capabilities of SOTA open source models while achieving competitive performance in omni interaction. Gander also demonstrates robustness in challenging real-world scenarios, including background noise interference, multi-party interactions, and backchannel communication. We release Gander together with its models, code, and data to facilitate further research and development in the community.