Emotion Recognition Using Continuous Wavelet Transform and Ensemble of Convolutional Neural Networks through Transfer Learning from Electroencephalogram Signal

Sara Bagherzadeh; Keivan  Maghooli; Ahmad  Shalbaf; Arash  Maghsoudi

doi:10.18502/fbt.v10i1.11512

Sara Bagherzadeh Department of Biomedical Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran
Keivan Maghooli Department of Biomedical Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran
Ahmad Shalbaf Department of Biomedical Engineering and Medical Physics, School of Medicine, Shahid Beheshti University of Medical Sciences, Tehran, Iran
Arash Maghsoudi Department of Biomedical Engineering, Science and Research Branch, Islamic Azad University, Tehran, Iran

DOI: https://doi.org/10.18502/fbt.v10i1.11512

Keywords: Emotion Recognition; Electroencephalogram; Deep Learning; Transfer Learning; Ensemble Approach; Continuous Wavelet Transform.

Abstract

Purpose: Emotions are integral brain states that can influence our behavior, decision-making, and functions. Electroencephalogram (EEG) is an appropriate modality for emotion recognition since it has high temporal resolution and is a non-invasive and cheap technique.

Materials and Methods: A novel approach based on Ensemble pre-trained Convolutional Neural Networks (ECNNs) is proposed to recognize four emotional classes from EEG channels of individuals watching music video clips. First, scalograms are built from one-dimensional EEG signals by applying the Continuous Wavelet Transform (CWT) method. Then, these images are used to re-train five CNNs: AlexNet, VGG-19, Inception-v1, ResNet-18, and Inception-v3. Then, the majority voting method is applied to make the final decision about emotional classes. The 10-fold cross-validation method is used to evaluate the performance of the proposed method on EEG signals of 32 subjects from the DEAP database.

Results:.The experiments showed that applying the proposed ensemble approach in combinations of scalograms of frontal and parietal regions improved results. The best accuracy, sensitivity, precision, and F-score to recognize four emotional states achieved 96.90% ± 0.52, 97.30 ± 0.55, 96.97 ± 0.62, and 96.74 ± 0.56, respectively.

Conclusion: So, the newly proposed model from EEG signals improves recognition of the four emotional states in the DEAP database.