Integration and generalization of kappas for multiple raters

J. A. Cohen's kappa (1960) for measuring agreement between 2 raters, using a nominal scale, has been extended for use with multiple raters by R. J. Light (1971) and J. L. Fleiss (1971). In the present article, these indices are analyzed and reformulated in terms of agreement statistics based on all pairs of raters. It has been argued that simultaneous agreement among all raters could provide an alternative basis for measuring multiple-rater agreement; however, agreement among raters can actually be considered to be an arbitrary choice along a continuum ranging from agreement for a pair of raters to agreement among all raters. Using this generalized concept of g -wise agreement, multiple-rater kappas are extended, interrelated, and illustrated. (4 ref)