Setting Standards for Judging the Rapid Film Reporting Section in Postgraduate Radiology Assessment: a Feasibility Study

Objective We investigated the feasibility of applying a standard-setting procedure for the rapid film reporting examination of the Malaysian National Conjoint Board of Radiology. Methods We selected the modified Angoff standard-setting process. Judges were nominated and trained, performance categories were discussed, and judges’ ratings on films were collected after an iterative procedure. The process was then evaluated for evidence of validity. Results A cut-off score of 92% resulted, compared with the 80% usually used. Judges were satisfied with the training and understood the procedure and their roles. In all, 27.3% felt that time given for the task was not sufficient. A total of 54.5% of judges thought the final passing cut-off score of 92% was too high, and 27.3% were not confident regarding its appropriateness. The inter-rater reliability was 0.928. External comparison with a ‘gold standard’ of supervisor ratings revealed a sensitivity of 0.25 and specificity of 1.00 compared with the traditional cut-off score having a sensitivity of 0.92 and specificity of 0.33. In this kind of situation, high specificity is considered to be more important than high sensitivity. Conclusion Standard setting for the rapid film reporting examination using the modified Angoff method was feasible with robust procedural, internal, and external validity evidence. Areas for improvement were identified to address the perceived high cut-off score obtained and improve the overall process.