Complex sparse spatial filter for decoding mixed frequency and phase coded steady-state visually evoked potentials

•A nobel command detection methods of SSVEP-BCI is proposed for mixed FPC visual stimuli.•EEG data with Sixteen target stimuli are used to evaluate the effectiveness of CSSF methods.•The proposed CSSF significantly outperform conventional FPC methods.•Our study confirms the proposed method is promising for implementing SSVEP-BCI with reduce visual fatigue. Mixed frequency and phase coding (FPC) can achieve the significant increase of the number of commands in steady-state visual evoked potential-based brain-computer interface (SSVEP-BCI). However, the inconsistent phases of the SSVEP over channels in a trial and the existence of non-contributing channels due to noise effects can decrease accurate detection of stimulus frequency. We propose a novel command detection method based on a complex sparse spatial filter (CSSF) by solving ℓ1- and ℓ2,1-regularization problems for a mixed-coded SSVEP-BCI. In particular, ℓ2,1-regularization (aka group sparsification) can lead to the rejection of electrodes that are not contributing to the SSVEP detection. A calibration data based canonical correlation analysis (CCA) and CSSF with ℓ1- and ℓ2,1-regularization cases were demonstrated for a 16-target stimuli with eleven subjects. The results of statistical test suggest that the proposed method with ℓ1- and ℓ2,1-regularization significantly achieved the highest ITR. The proposed approaches do not need any reference signals, automatically select prominent channels, and reduce the computational cost compared to the other mixed frequency-phase coding (FPC)-based BCIs. The experimental results suggested that the proposed method can be usable implementing BCI effectively with reduce visual fatigue.