Spatio-time-frequency joint sparse optimization with transfer learning in motor imagery-based brain-computer interface system

•The proposed algorithm optimized parameters simultaneously in spatial domain, time domain and frequency domain with ABC and LASSO.•The instance-based transfer learning was adopted to improve the classification accuracy of MI EEG in the case of less labeled target data.•The experimental results revealed that the performance of the proposed algorithm was better than that of other algorithms for three datasets.•The conclusion was that STFSTL was a robust algorithm which can achieve high classification accuracy in a small training set. Motor imagery-based brain-computer interface (MI-BCI) is widely considered as the most promising BCI. Non-stationary of EEG data and long BCIs’ calibration time are main problems that affect the practicability of MI-BCI. In this paper, we propose a new algorithm, i.e. spatio-time-frequency joint sparse optimization algorithm with transfer learning (STFSTL) to achieve satisfactory classification accuracy with small training set. By introducing artificial bee colony (ABC) algorithm and least absolute shrinkage and selection operator (LASSO), the algorithm optimized parameters in spatial domain, time domain and frequency domain simultaneously. The similarity between data was measured by Euclidean distance. Through instanced-based transfer learning, the source data which was most similar to the target data was selected as the auxiliary data to train the target classifier. We evaluated the performance of the proposed algorithm on three data sets, including a private data set and two public data sets. The classification accuracy of the proposed algorithm with one fifth of the training data was higher than that of five other algorithms. Paired t-test analysis revealed that the accuracy of STFSTL and that of five other algorithms were significantly different. The experimental results suggested that the proposed algorithm with less target data can effectively achieve higher classification accuracy than traditional algorithms. It’s likely to have a broad application prospect in MI-BCI.