Driver fatigue detection method based on temporal–spatial adaptive networks and adaptive temporal fusion module

The reduction of traffic accidents by determining the driver’s state through fatigue detection is a worthy research issue. Most of the current fatigue driving detection method fails to fully utilize the temporal features of fatigue. To address this problem, this paper proposes a driver fatigue detection method combined with temporal–spatial adaptive networks (TSNet) and adaptive temporal fusion module (ATFM). First, a frame sequence of T frames is obtained by strided sampling of the input video and data enhancement. Subsequently, the temporal–spatial adaptive module (TSAM) is used as the core module and incorporated into Efficientnet-v2 to construct TSNet, which adaptively extracts temporal features according to different videos, adds attention weights to discriminative spatial and channel features, and fully extracts fatiguing temporal–spatial features of videos. Finally, ATFM is utilized to learn the weights between the fatigue classification scores of each frame in the frame sequence and adaptively fuses the fatigue classification scores of individual frames to obtain fatigue prediction results, increasing the extent of the influence of keyframes on the fatigue prediction results. In this paper, the proposed method achieves an accuracy of 89.42% on the NTHU-DDD dataset, which is better than other state-of-the-art methods, and the number of parameters of the proposed method is 24.70M, which is smaller than most of the methods. Through a series of comparative experiments, TSNet and ATFM alone also outperform models and modules with similar functionality. •Propose a fatigue detection method based on TSNet-ATFM using video inputs for lack of temporal characterization in fatigue detection.•The TSAM is designed as the core module and combined with Efficientnet-v2 to construct feature extraction network TSNet.•The ATFM is designed to handle the fatigue classification scores of each frame and increase the influence of keyframes on the results.