Mixed Standard Scale Response Inconsistencies as Reliability Indices

The mixed standard scale is a rating format which allows researchers to count illogical (i.e., internally inconsistent) response patterns. These counts have been interpreted as an index of rater reliability, based on the notion that illogical response patterns reflect carelessness in responding. This study investigated the meaning of these counts, using 943 accountants as raters. It was found that the counts of illogical response patterns were not related to reliability as measured by Cronbach's alpha, and alternative interpretations for such indices are suggested.