CineBrain：自然情境下视听叙事处理的大规模多模态脑数据集

CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic Audiovisual Narrative Processing

March 10, 2025

作者: Jianxiong Gao, Yichang Liu, Baofeng Yang, Jianfeng Feng, Yanwei Fu

cs.AI

摘要

在本論文中，我們介紹了CineBrain，這是首個在動態視聽刺激下同步記錄腦電圖（EEG）和功能性磁共振成像（fMRI）的大規模數據集。認識到EEG的高時間分辨率與fMRI的深層腦部空間覆蓋的互補優勢，CineBrain為六名參與者提供了來自熱門電視劇《生活大爆炸》的約六小時敘事驅動內容。基於這一獨特數據集，我們提出了CineSync，這是一種創新的多模態解碼框架，它將多模態融合編碼器與基於擴散的神經潛在解碼器相結合。我們的方法有效地融合了EEG和fMRI信號，顯著提升了複雜視聽刺激的重建質量。為了促進嚴謹的評估，我們引入了Cine-Benchmark，這是一個全面的評估協議，從語義和感知維度評估重建效果。實驗結果表明，CineSync在視頻重建性能上達到了最先進的水平，並凸顯了我們在結合fMRI和EEG重建視頻和音頻刺激方面的初步成功。項目頁面：https://jianxgao.github.io/CineBrain。

English

In this paper, we introduce CineBrain, the first large-scale dataset featuring simultaneous EEG and fMRI recordings during dynamic audiovisual stimulation. Recognizing the complementary strengths of EEG's high temporal resolution and fMRI's deep-brain spatial coverage, CineBrain provides approximately six hours of narrative-driven content from the popular television series The Big Bang Theory for each of six participants. Building upon this unique dataset, we propose CineSync, an innovative multimodal decoding framework integrates a Multi-Modal Fusion Encoder with a diffusion-based Neural Latent Decoder. Our approach effectively fuses EEG and fMRI signals, significantly improving the reconstruction quality of complex audiovisual stimuli. To facilitate rigorous evaluation, we introduce Cine-Benchmark, a comprehensive evaluation protocol that assesses reconstructions across semantic and perceptual dimensions. Experimental results demonstrate that CineSync achieves state-of-the-art video reconstruction performance and highlight our initial success in combining fMRI and EEG for reconstructing both video and audio stimuli. Project Page: https://jianxgao.github.io/CineBrain.