Practical Strategies for Dealing with Unreliability in Competency Assessments

In many practical settings, test results are used as a basis for making decisions of a categorical nature. Students are judged to have mastery if they score above some predetermined cutoff score. Teachers are eligible for merit pay if they score above some designated point on a performance measure. Students are allowed to graduate from high school if they demonstrate minimum math and language levels on a functional test. In these and numerous other examples, decisions are made based on assessment devices that are known to be less than perfectly reliable. For most examinees, the unreliability of a well-designed assessment device is not sufficient to lead to misclassification. For a significant portion of examinees who have scores near the cutting score, however, the risk of misclassification is substantial even for instruments with relatively high retest reliability. Several practical suggestions are made for minimizing the risk of misclassification of examinees with scores near the cutoff score.