Character encoding based on occurrence probability enhances the performance of SSVEP-based BCI spellers

•The 1–2 hierarchical structure provides selecting most used characters in 1 step.•Assigning easier code to more commonly used characters increases the ITR.•The proposed character encoding leads the system becomes more user-friendly.•We estimate the ITR accurately using the definition based on symbol probability.•Accurate estimation of the ITR leads the system evaluation becomes more reliable. Steady-state visual evoked potential (SSVEP) is a control signal which is widely used in brain-computer interface (BCI) systems. The SSVEP-based spellers with hierarchical structure have a limitation of low ITR. To improve the ITR in these spellers, we effectively applied the character encoding based on the character frequency rate. We proposed the 1–2 level hierarchical structure that allows the user to spell the most used characters just in one stage, while other characters will be selected through two stages. We also considered the latency at the start of each trial, to enhance the SSVEP classification accuracy. To estimate the ITR more accurately, we used a novel ITR definition for the first time, which considers the symbol occurrence probability. The proposed speller achieved the mean classification accuracy of 90.5%, the ITR of 48.3 bit/min, and the speed of 13.2 char/min. The latency varies for different subjects, and the mean value of 0.2 was determined across all individuals. Considering the character encoding enhances the performance of SSVEP-based BCI spellers. The proposed speller provides a reliable and easy-to-use assistive communication system for locked-in patients.