The Validity of Reliability Measures

It is shown that some commonly used indices can be misleading in their quantification of reliability. The effects are seen most vividly in assessing the reliability of gain or difference scores. In addition, it is explained that the calculation of all the currently used indices is based on a wrong assumption about the value of the correlation coefficient between true and error scores being zero. The practical effects of using this wrong assumption are shown to be very significant. Proposals are made to avoid both these sources of invalidity using a procedure to assess reliability in terms of upper and lower limits for the true scores of each examinee.