Reliability of a Minimal Competency Score for an Annual Skills Mastery Assessment

Objective. To determine whether the modified Angoff process can be used to calculate a reliable minimal competency (“cut”) score for the Annual Skills Mastery Assessment (ASMA). Methods. Three panels of pharmacy faculty members used a modified Angoff method to create a minimal competency score for 60 previously used test items. The panels did not know which items were included. Data were analyzed to determine differences between rating sessions, faculty type, item difficulty, and rater scoring bias. Results. The cut score generated was not significantly different by session or faculty type. The range of cut scores varied by less than 3% per examination. Faculty panelists correctly predicted student performance on items grouped as easy, medium, and hard. Conclusion. A properly constructed faculty panel can determine a reliable cut score and accurately rank relative test item difficulty using the modified Angoff process.