Data-driven learning fatigue detection system: A multimodal fusion approach of ECG (electrocardiogram) and video signals

•A novel multiple classifier capable of detecting learning fatigue in daily life (without specific stimulations).•A multimodal approach with video and ECG signals.•A hybrid of handcrafted and deep learning features.•Confirmation of the performance based on 10-fold cross-validation.•Achieving a detection accuracy of 99.6% on one public database.•Achieving a detection accuracy of 91.8% on the self-collected database. Fatigue could lead to low efficiency and even serious disaster. In the educational field, detecting fatigue could help adjust teaching strategies accordingly when a student is inactive, which can potentially improve learning efficiency. Despite numerous studies in fatigue detection, there is still a lack of multiple classifier systems capable of detecting fatigue in daily life (without specific stimulations). To initially alleviate this problem, this study develops a learning fatigue detection system using a multimodal approach with ECG and video signals, classifying a learner’s state into three categories: alert, normal, and fatigued. To validate performance, the proposed system is tested on (i) an open-source dataset DROZY (n = 35) and (ii) a self-collected dataset captured in a learning environment (n = 92). The experimental results based on 10-fold cross-validation demonstrate that the system outperforms the state-of-the-art approaches, achieving a detection accuracy of 99.6% and 91.8% on the two datasets, respectively.