E-FNet: A EEG-fNIRS dual-stream model for Brain–Computer Interfaces

With the rapid advancements in Brain–Computer Interfaces (BCI) and multimodal technology, researchers are actively exploring the intricate domain of multimodal BCI. The integration of electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) in BCI systems has garnered significant attention due to the combination of strengths from both modalities and the overcoming of limitations inherent in standalone systems. In this study, the E-FNet architecture is introduced, a (2+1)D dual-stream model explicitly designed to process multimodal EEG and fNIRS data for brain–machine interface applications. By leveraging the (2+1)D architecture and adopting a bias-free design, E-FNet efficiently processes multimodal data. A unified 4D tensor representation enables the achievement of both temporal and spatial alignment of EEG and fNIRS signals. For Motor Imagery (MI) and Mental Arithmetic (MA) tasks, the early integration of EEG and fNIRS signals yields remarkable accuracy rates of 95.86% and 95.80% respectively, outperforming the accuracy obtained with EEG signals alone. Moreover, the minimal variability in results across subjects underscores the complementary effects achieved through the integration of different signals. •Improved accuracy and robustness through EEG and fNIRS signal synergy.•Enhanced EEG and fNIRS analysis leveraging channel topological relationships.•E-FNet aligns EEG and fNIRS signals in time and space using a unified 4D tensor.•Using (2+1)D dual-stream model boosts efficiency and enhances nonlinear capabilities.