Interactive triage simulator revealed important variability in both process and outcome of emergency triage

(1) to evaluate the performance of emergency department triage; (2) to explore the variability of the triage process; and (3) to examine the reliability of a four-level triage scale, using an interactive triage simulator. We developed 22 interactive computerized vignettes describing patients presenting at the Emergency Department. Each vignette displayed the presenting complaint and offered the possibility to ask questions and obtain vital signs before deciding on the triage severity rating. The vignettes were rated twice by 45 nurses and 8 physicians. (1) The concordance between the observed triage decision and an expert-attributed emergency level was perfect in 58% of the situations. Triage acuity was overestimated in 11%, and underestimated in 31%. (2) There was a wide variability in the triage process across observers and vignettes. The mean number of questions varied from 1.77 to 18.95 across individuals, and from 3.96 to 11.60 across vignettes. (3) Finally, the test–retest reliability of our instrument was good (weighted kappa = 0.82) but the interrater reliability was moderate (weighted kappa = 0.41). The computerized triage simulator is an innovative tool to evaluate the process and the performance of triage and to evaluate the reliability of a triage instrument.