BrainJanus：跨大脑、视觉与语言的统一理解与生成模型

摘要

建模外部感官刺激与内部神经活动之间的双向对应关系已成为神经科学的关键前沿。然而，现有方法主要将大脑编码与解码视为孤立任务，过度依赖单模态对齐和外部先验，忽视了大脑作为多模态整合系统的内在本质。为解决这些局限，我们提出BrainJanus——首个将大脑、视觉与语言统一于单一框架的脑模型。具体而言，我们引入统一脑分词器（Unified Brain Tokenizer），将连续神经动态量化为离散标记，使其在共享的全域空间中与视觉和语言表征对齐。在此基础上，我们采用全能自回归架构（All-in-One），利用下一标记预测实现无缝的任意到任意生成，涵盖图像到大脑与文本到大脑的编码，以及大脑到图像与大脑到文本的解码。大量实验表明，BrainJanus在多个基准测试中均取得卓越性能。此外，我们的框架展现出零样本泛化能力，并保留了可解释的生物学拓扑结构，凸显其作为通用脑建模范式的潜力。代码已开源至 https://github.com/HaitaoWuTJU/BrainJanus{GitHub}。

English

Modeling the bidirectional correspondence between external sensory stimuli and internal neural activity has emerged as a critical frontier in neuroscience. However, existing approaches predominantly treat brain encoding and decoding as isolated tasks, relying heavily on unimodal alignment and external priors while overlooking the brain's intrinsic nature as a multimodal integration system. To address these limitations, we propose BrainJanus, the first unified brain model that integrates brain, vision, and language within a single framework. Specifically, we introduce a Unified Brain Tokenizer to quantize continuous neural dynamics into discrete tokens aligned with visual and linguistic representations in a shared Omni space. Building on this, we utilize an All-in-One autoregressive architecture that leverages next-token prediction to enable seamless any-to-any generation, which encompasses image-to-brain and text-to-brain encoding, and brain-to-image and brain-to-text decoding. Extensive experiments demonstrate that BrainJanus achieves superior performance across diverse benchmarks. Furthermore, our framework exhibits zero-shot generalization and preserves interpretable biological topography, highlighting its potential as a general-purpose brain modeling paradigm. The code is available at https://github.com/HaitaoWuTJU/BrainJanus{GitHub}.