Difficulty and Discriminative Ability of Various Categories of Multiple Choice Questions in Medical School Preclerkship Examinations

Background In the USA, medical licensing exams have attempted to test the higher levels of Bloom’s taxonomy by using multiple choice questions (MCQ) that incorporate clinical or experimental vignettes. Aim We wished to determine if the nature of the vignette altered the difficulty and discriminative ability of our MCQ. Methods We assigned our regular course MCQ to categories. Recall questions had no context. Pseudo-vignette questions had a context not needed to answer the question. Vignette 1 or vignette 2 questions required extracting information from the vignette. Difficulty (DIFF), discrimination index (DI), and point biserial correlation coefficient (PtBiS) were calculated for 1,690 questions used during our preclerkship curriculum. Results Recall questions were more common during the first year. Pseudo-vignette questions maintained approximately the same frequency across the courses. Vignette 1 and vignette 2 questions increased gradually. During the first year, questions were less difficult but more discriminating than during the second year. In the first year, recall questions were less difficult and discriminating than vignette questions. These differences disappeared during the second year, when DIFF, DI, and PtBiS did not differ significantly among the question formats. Conclusions The differences between the results of testing in first and second years may reflect experience with vignette-type questions, true learning, or a combination of both.