A Novel Object Categorization Decoder from fMRI Signals Using Deep Neural Networks

Alireza  Shakeripour; Zahra  Bahmani; Poorya  Aghaomidi; Shima  Seyed-Allaei

doi:10.18502/fbt.v12i3.19186

Alireza Shakeripour Department of Biomedical Engineering, Faculty of Electrical & Computer Engineering, Tarbiat Modares University, Tehran, Iran
Zahra Bahmani Department of Biomedical Engineering, Faculty of Electrical & Computer Engineering, Tarbiat Modares University, Tehran, Iran
Poorya Aghaomidi Department of Biomedical Engineering, Faculty of Electrical & Computer Engineering, Tarbiat Modares University, Tehran, Iran
Shima Seyed-Allaei School of Cognitive Sciences, Institute for Research in Fundamental Sciences, Tehran, Iran

DOI: https://doi.org/10.18502/fbt.v12i3.19186

Keywords: Functional Magnetic Resonance Imaging; Deep Learning; Object Recognition; Region Of Interest Connectivity; Brain Decoding.

Abstract

Purpose: Understanding neural mechanisms is critical for discerning the nature of brain disorders and enhancing treatment methodologies. Functional Magnetic Resonance Imaging (fMRI) plays a vital role in gaining this knowledge by recording various brain regions. In this study, our primary aim was to categorize visual objects based on fMRI data during a natural scene viewing task. We intend to elucidate the challenges and limitations of previous models in order to produce a generalizable model across different subjects using advanced deep-learning methods.

Materials and Methods: We've designed a new deep-learning model based on transformers for processing fMRI data. The model includes two blocks, the first block receives fMRI data as input and transforms the input data to a set of features called fMRI space. Simultaneously a visual space is extracted from visual images using a pre-trained inceptionv3 network. The model tries to construct the fMRI space similar to the extracted visual space. The other block is a Fully Connected (FC) network for object recognition based on fMRI space. Using transformer capabilities and an overlapping method, the proposed architecture accounts for structural changes across different voxel sizes of the subjects' brains.

Results: A unique model was trained for all subjects with different brain sizes. The results demonstrated that the proposed network achieves an impressive similarity correlation between visual space and fMRI space around 0.86 for train and 0.86 for test dataset. Furthermore, the classification accuracy was about 70.3%. These outcomes underscored the effectiveness of our fMRI transformer network in extracting features from fMRI data.

Conclusion: The results indicated the potential of our model for decoding images from the brain activities of new subjects. This unveils a novel direction in image reconstruction from neural activities, an area that has remained relatively uncharted due to its inherent intricacies.