BrainJanus：一個跨大腦、視覺與語言的統一理解與生成模型

摘要

外在感覺刺激與內在神經活動之間的雙向對應建模已成為神經科學的關鍵前沿。然而，現有方法多將腦編碼與解碼視為孤立任務，高度依賴單模態對齊與外部先驗，忽略了腦作為多模態整合系統的本質。為解決這些局限性，我們提出BrainJanus，首個在單一架構中整合腦、視覺與語言的統一腦模型。具體而言，我們引入統一腦分詞器（Unified Brain Tokenizer），將連續神經動態量化為離散標記，並在共享的全空間（Omni space）中與視覺及語言表徵對齊。在此基礎上，我們利用全合一（All-in-One）自回歸架構，透過下一個標記預測實現無縫的任意到任意生成，涵蓋影像到腦與文字到腦的編碼，以及腦到影像與腦到文字的解碼。廣泛實驗表明，BrainJanus在各種基準測試中均達到優異性能。此外，本框架展現出零樣本泛化能力，並保留可解釋的生物拓撲結構，凸顯其作為通用腦建模範式的潛力。程式碼已於 https://github.com/HaitaoWuTJU/BrainJanus{GitHub} 開源。

English

Modeling the bidirectional correspondence between external sensory stimuli and internal neural activity has emerged as a critical frontier in neuroscience. However, existing approaches predominantly treat brain encoding and decoding as isolated tasks, relying heavily on unimodal alignment and external priors while overlooking the brain's intrinsic nature as a multimodal integration system. To address these limitations, we propose BrainJanus, the first unified brain model that integrates brain, vision, and language within a single framework. Specifically, we introduce a Unified Brain Tokenizer to quantize continuous neural dynamics into discrete tokens aligned with visual and linguistic representations in a shared Omni space. Building on this, we utilize an All-in-One autoregressive architecture that leverages next-token prediction to enable seamless any-to-any generation, which encompasses image-to-brain and text-to-brain encoding, and brain-to-image and brain-to-text decoding. Extensive experiments demonstrate that BrainJanus achieves superior performance across diverse benchmarks. Furthermore, our framework exhibits zero-shot generalization and preserves interpretable biological topography, highlighting its potential as a general-purpose brain modeling paradigm. The code is available at https://github.com/HaitaoWuTJU/BrainJanus{GitHub}.