The unilateral upper limb classification from fMRI-weighted EEG signals using convolutional neural network

•The distinguishability and influence with motor areas of the brain are validated by functional Magnetic Resonance Imaging (fMRI) statistical analysis and Electroencephalogram (EEG) classification.•We propose the fMRI-weighted Convolutional Neural Network (CNN) classification based on fMRI statistical analysis. The fMRI-weighted CNN shows better performance on multiple tasks motor imagery classification.•The same activation features for each task are found in fMRI and EEG visualizations results. Unilateral upper limb multitasking brings essential improvements to stroke rehabilitation and prosthetic control. However, the influence and recognition of multiple tasks are core issues in the Motor Imagery (MI) system. First, we design the unilateral upper limb MI experimental paradigm and acquire asynchronous functional Magnetic Resonance Imaging (fMRI) and Electroencephalogram based on MI (MI-EEG) data. Then, Brain activation areas for each task are statistically analyzed by fMRI data. A novel fMRI-weighted Convolutional Neural Network (CNN) is designed to reassign each channel’s weight based on brain activation areas to improve classification accuracy. Finally, the EEG data is classified by the fMRI-weighted CNN. The four classes of MI tasks reflect significant activation in brain motor areas. The average classification accuracy of fMRI-weighted CNN is 47.0%. The visualization results show the similarity between fMRI and EEG activation areas in the same task. The distinguishability of multiple tasks and the influence on motor areas of the brain are confirmed by fMRI experiments. The classification accuracy of multitasks is improved according to the fMRI-weighted CNN. This paper provides a reference for further research into asynchronous fMRI-EEG modeling and multitask MI classification.