Composite Reliability of Multilevel Data: It's About Observed Scores and Construct Meanings

This article shows how the concept of reliability of composite scores, as defined in classical test theory, can be extended to the context of multilevel modeling. In particular, it discusses the contributions and limitations of the various level-specific reliability indices proposed by Geldhof, Preacher, and Zyphur (2014), denoted as ω̃b and ω̃w (and also αãb and α̃w). One major limitation of those indices is that they are quantities for latent, unobserved level-specific composite scores, and are not suitable for observed composites at different levels. As illustrated using simulated data in this article, ω̃b can drastically overestimate the true reliability of between-level composite scores (i.e., observed cluster means). Another limitation is that the development of those indices did not consider the recent conceptual development on construct meanings in multilevel modeling (Stapleton & Johnson, 2019; Stapleton, Yang, & Hancock, 2016). To address the second limitation, this article defines reliability indices (ω2l, ωb, ωw, α2l, αb, αw) for three types of multilevel observed composite scores measuring various multilevel constructs: individual, configural, shared, and within-cluster. The article also shows how researchers can obtain sample point and interval estimates using the derived formulas and the provided R and Mplus code. In addition, a large-scale national data set was used to illustrate the proposed methods for estimating reliability for the three types of multilevel composite scores, and practical recommendations on when different indices should be reported are provided. Translational Abstract Reporting of reliability information of measures on the studied sample is a basic requirement in psychological research. For multilevel data-data that have a hierarchical structure such as students nested within schools and survey participants nested within neighborhoods, previous research commonly used the between-level and within-level composite reliability proposed by Geldhof, Preacher, and Zyphur (2014). However, this article shows that the previously proposed between-level composite reliability can provide overly optimistic reliability coefficient because it ignores one major source of error, namely the sampling error of cluster means. To obtain more accurate reliability information for multilevel data, this article proposes alternative reliability indices that correctly account for the different sources of measurement error. The computation of the bet