The cognitive process of test takers when using the script concordance test rating scale

Context Clinical decision making (CDM) skills are important to learn and assess in order to establish competence in trainees. A common tool for assessing CDM is the script concordance test (SCT), which asks test takers to indicate how a new clinical finding influences a proposed plan using a Likert‐type scale. Most criticisms of the SCT relate to its rating scale but are largely theoretical. The cognitive process of test takers when selecting their responses using the SCT rating scale remains understudied, but is essential to gathering validity evidence for use of the SCT in CDM assessment. Methods Cases from an SCT used in a national validation study were administered to 29 residents and 14 staff surgeons. Semi‐structured cognitive interviews were then conducted with 10 residents and five staff surgeons based on the SCT results. Cognitive interview data were independently coded by two data analysts, who specifically sought to elucidate how participants mapped their internally generated responses to any of the rating scale options. Results Five major issues were identified with the response matching cognitive process: (a) the meaning of the ‘0' response option; (b) which response corresponds to agreement with the planned management; (c) the rationale for picking ‘±1' versus ‘±2'; (d) which response indicates the desire to undertake the planned management plus an additional procedure, and (e) the influence of time on response selection. Conclusions Studying how test takers (experts and trainees) interpret the SCT rating scale has revealed several issues related to inconsistent and unintended use. Revising the scale to address the variety of interpretations could help to improve the response process validity of the SCT and therefore improve the SCT's ability to be used in CDM skills assessments. The authors studied how test‐takers (experts and trainees) understand and interpret the SCT rating scale and demonstrated several issues related to inconsistent and unintended use.