Usability and reproducibility of three tools to assess medical students and residents in emergency medicine

Objectives It is critical to assess competency of medical students and residents in emergency medicine (EM) during undergraduate and graduate medical education. However, very few valid tools exist to assess both technical and nontechnical skills in the specific context of EM. Three Acute Care Assessment Tools (ACAT 1, 2, and 3) have been previously developed for three acute care conditions: cardiac arrest (1), coma (2), and acute respiratory failure (3). This study aimed to evaluate the reproducibility of the tools. Methods The tool was tested using recorded videos of simulation sessions of fourth year medical students and first year residents in EM. Raters independently reviewed the videos two times in a 3‐month interval, and interrater and intrarater reliability using intraclass correlation (ICC) was calculated. Secondary endpoints included the completeness rate and relevance of the ACAT. Results Sixty‐two sessions were recorded and 48 videos analyzed (18 for CA and 15 for both respiratory failure and coma. The learners were residents in 32 (66%) of videos. Interrater reliability was excellent (ICC >0.9 for all three contexts) and so was the intrarater reliability (>0.88), both upon first review (month 0, M0) and at 3 months (M3). The usability of the ACAT was good, with a completeness of the items that ranged from 96% to 100%. Only one item of the ACAT 1 had a relevance of 27%, as it could not be completed in 13 scenarios out of 18. Conclusions The results demonstrate educators can evaluate students similarly utilizing video recordings of simulated medical scenario. The excellent completeness of the rated items advocated for good usability. The three ACATs can be utilized to assess for completeness of predefined tasks in three acute care broad scenario in a competency‐based medical education framework.