CoSleepNet: Automated sleep staging using a hybrid CNN-LSTM network on imbalanced EEG-EOG datasets

•A new deep learning model with minimal model complexity is presented for sleep staging using multi-channel sleep data.•The proposed model aims to increase the classification performance in unbalanced data.•Focal Loss is used to take the place of the traditional categorical cross-entropy loss function.•The performance of the proposed method is promising compared to existing ones. Sleep relaxes and rests the body by slowing down the metabolism, making us physically stronger and fitter when we wake up. However, in a sleep disorder that may occur in humans, this process is reversed and various disorders occur in the body. Therefore, determining sleep stages is vital for diagnosing and treating such sleep disorders. However, manual scoring of sleep stages is tedious, time-consuming and requires considerable expertise. It also suffers from inter-observer variability. Deep learning techniques can automate this process, overcome these problems and produce more consistent results. This study proposes a new hybrid neural network architecture using focal loss and discrete cosine transform methods to solve the training data imbalance problem. The model was trained on four different databases using k-fold validation strategies (subject-wise), and the highest score was 87.11% accuracy, 81.81% Kappa score, and 79.83% MF1 when using two channels (EEG-EOG). The results of our approach are promising when compared to existing methods.