Measuring the impact of simulation debriefing on the practices of interprofessional trauma teams using natural language processing

Natural language processing (NLP) may be a tool for automating trauma teamwork assessment in simulated scenarios. Using the Trauma Nontechnical Skills Assessment (T-NOTECHS), raters assessed video recordings of trauma teams in simulated pre-debrief (Sim1) and post-debrief (Sim2) trauma resuscitations. We developed codes through directed content analysis and created algorithms capturing teamwork-related discourse through NLP. Using a within subjects pre-post design (n = 150), we compared changes in teams' Sim1 versus Sim2 T-NOTECHS scores and automatically coded discourse to identify which NLP algorithms could identify skills assessed by the T-NOTECHS. Automatically coded behaviors revealed significant post-debrief increases in teams' simulation discourse: Verbalizing Findings, Acknowledging Communication, Directed Communication, Directing Assessment and Role Assignment, and Leader as Hub for Information. Our results suggest NLP can capture changes in trauma team discourse. These findings have implications for the expedition of team assessment and innovations in real-time feedback when paired with speech-to-text technology. •Natural Language Processing techniques can detect changes in trauma teams' behavior.•Both derived and naïve algorithms were able to detect changes in discourse patterns.•Not all automated performance measures showed improvement captured by the T-NOTECHS.