Accuracy of Option Elimination Prediction: Effect of a Technique To Improve Rater Agreement with Empirical Data

The effects of feedback about the ratings of other judges on subsequent ratings using the Nedelsky method and the ability of judges to retain or eliminate options in a manner consistent with the judgments of minimally competent examinees were studied using data from a basic algebra examination administered to 227 college students in 1987. The instrument used included 35 four-option multiple-choice items and was administered to the subjects at the start of an introductory statistics course to identify those in need of remedial instruction. Seven experienced instructors (judges) provided item ratings. Feedback did have an effect. Correlations between judges' estimates of item difficulty and the actual item difficulty increased slightly on the second rating. Improvements in both correlations and the accuracy of option elimination suggest that iterative procedures may be of value with the Nedelsky method. The requirement that judges evaluate individual options in a Nedelsky procedure can lead to accurate assessments of the plausibility of options. Nine tables present the study results. (SLD)