ASSESSING LEARNING ACROSS TRIALS IN THE AIR TRAFFIC SCENARIOS TEST (ATST)

Introduction: The Air Force Research Laboratory (AFRL) requested that the Federal Aviation Administration (FAA) examine the Air Traffic Scenarios Test (ATST) to determine if it could be shortened while maintaining reliability and validity. The ATST is one of 12 subtests in the FAA's Air Traffic Selection and Training Battery (AT-SAT). When combined with the Armed Services Vocational Aptitude Battery (ASVAB), the ATST added incremental validity to the prediction of controller training success. However, the current version of ATST was too long to be included as a permanent addition to the ASVAB. We examined ATST trial-level data to determine if any trials could be dropped, to shortening the ATST testing time. Method: Data are from 391 job applicants who took AT-SAT operationally during a two-month period. We examined scores for efficiency of aircraft movement to destination, procedural accuracy, and maintenance of aircraft separation. A multivariate repeated measures design with performance variables and trials as within subject factors was conducted. Wilk's Lambda was significant for the effect of trials. Univariate ANOVAs showed that the effect of trials was consistent across the performance variables. Results: On all measures, performance improved from Trial 1 to Trial 2. Performance declined on Trial 3, most likely due to the 30% increase in the number of aircraft managed during the scenario. The large improvement in performance from Trial 1 to Trial 2 is probably due to insufficient practice time provided by the three practice trials. Conclusions: It is possible that, with a more structured approach to designing practice sessions, one of the first two trials may be replaced, thus reducing the total testing time. Future research should examine the reliability and validity of a shorter version of the ATST.