Prediction and extraction of tower controller commands for speech recognition applications

Air traffic controllers' (ATCos) workload often is a limiting factor for air traffic capacity. Thus, electronic support systems intend to reduce ATCos' workload. Automatic speech recognition can extract controller command elements from verbal clearances to deliver automatic input for air traffic control systems, thereby avoiding manual input. Assistant Based Speech Recognition (ABSR) with high command recognition rates and low error rates has proven to dramatically reduce ATCos’ workload and increase capacity in approach scenarios. However, ABSR needs accurate hypotheses on expected commands and accurate extractions of command annotations from utterance transcriptions to achieve the required performance. Based on the experience of implementation for approach control, a hypotheses generator and a command extractor have been developed for speech recognition applications regarding tower control communication to face current and future challenges in the aerodrome environment. Three human-in-the-loop multiple remote tower simulation studies were performed with 16 ATCos from Hungary, Lithuania, and Finland at DLR Braunschweig from 2017 to 2019. Roughly 100 h of speech with corresponding radar data were recorded. Around 6000 speech utterances resulting in 16,000 commands have been manually transcribed and annotated. Some parts of the data have been used for training prediction models and command extraction algorithms. Other parts were used for evaluation of command prediction and command extraction. The automatic command extractor achieved a command extraction rate of 96.7%. The hypotheses generator showed operational feasibility with a sufficiently low command prediction error rate of 7.3%. •Assistant Based Speech Recognition for Aerodrome Air Traffic Control.•First Implementation of Tower Command Hypotheses Generator and Command Extractor.•Multiple Remote Tower Trials with 16,000 commands from 16 Air Traffic Controllers.•Command Prediction Error Rate regarding Gold Annotations is 7.3%.•Command Extraction Rate of Annotations on Manual Transcriptions is 96.7%.