MindEye2：共享主体模型使fMRI到图像仅需1小时数据

MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data

March 17, 2024

作者: Paul S. Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A. Norman, Tanishq Mathew Abraham

cs.AI

摘要

大脑活动重建的视觉感知已经取得了巨大进展，但这些方法的实际效用却受到了限制。这是因为这些模型是针对每个受试者独立训练的，每个受试者需要数十小时昂贵的fMRI训练数据才能获得高质量的结果。本研究展示了仅使用1小时fMRI训练数据就能实现高质量重建。我们在7个受试者上预训练我们的模型，然后在新受试者上用少量数据进行微调。我们的新颖功能对齐程序将所有脑数据线性映射到一个共享主体的潜在空间，然后通过一个共享的非线性映射将其映射到CLIP图像空间。然后，我们通过微调Stable Diffusion XL来接受CLIP潜在空间而不是文本作为输入，将其从CLIP空间映射到像素空间。这种方法提高了在有限训练数据情况下跨受试者的泛化能力，并且与单受试者方法相比，实现了最先进的图像检索和重建指标。MindEye2展示了如何能够从一次MRI设施的访问中实现准确的感知重建。所有代码都可以在GitHub上找到。

English

Reconstructions of visual perception from brain activity have improved tremendously, but the practical utility of such methods has been limited. This is because such models are trained independently per subject where each subject requires dozens of hours of expensive fMRI training data to attain high-quality results. The present work showcases high-quality reconstructions using only 1 hour of fMRI training data. We pretrain our model across 7 subjects and then fine-tune on minimal data from a new subject. Our novel functional alignment procedure linearly maps all brain data to a shared-subject latent space, followed by a shared non-linear mapping to CLIP image space. We then map from CLIP space to pixel space by fine-tuning Stable Diffusion XL to accept CLIP latents as inputs instead of text. This approach improves out-of-subject generalization with limited training data and also attains state-of-the-art image retrieval and reconstruction metrics compared to single-subject approaches. MindEye2 demonstrates how accurate reconstructions of perception are possible from a single visit to the MRI facility. All code is available on GitHub.