AI‐based language tutoring systems with end‐to‐end automatic speech recognition and proficiency evaluation

This paper presents the development of language tutoring systems for non‐native speakers by leveraging advanced end‐to‐end automatic speech recognition (ASR) and proficiency evaluation. Given the frequent errors in non‐native speech, high‐performance spontaneous speech recognition must be applied. Our systems accurately evaluate pronunciation and speaking fluency and provide feedback on errors by relying on precise transcriptions. End‐to‐end ASR is implemented and enhanced by using diverse non‐native speaker speech data for model training. For performance enhancement, we combine semisupervised and transfer learning techniques using labeled and unlabeled speech data. Automatic proficiency evaluation is performed by a model trained to maximize the statistical correlation between the fluency score manually determined by a human expert and a calculated fluency score. We developed an English tutoring system for Korean elementary students called EBS AI PengTalk and a Korean tutoring system for foreigners called KSI Korean AI Tutor. Both systems were deployed by South Korean government agencies.