The reliability of composites of referee assessments of manuscripts

Hargens and Herting (1990) note that studies evaluating the reliability of referee assessments of manuscripts have underestimated the reliability of the review process for two reasons. First, past studies have assumed equal intervals between the categories used in assessing manuscripts (e.g., accept, accept after minor revision, revise and resubmit, and reject). Second, the procedures used have measured the reliability based on a single referee's evaluation for each manuscript rather than on a composite of two or more referees. This paper shows how to estimate the reliability based on one, two, or any number of referees, and we apply these procedures to estimate the reliability of journal reviews based on one, two, or three referees for each of the journals examined by Hargens and Herting.