I Only Have One Rater Per Ratee, So What? The Impact of Clustered Performance Rating Data on Operational Validity Estimates

Test validation is fundamental to industrial-organizational psychology and to many of the interventions that we use to improve organizational performance. Problems with performance ratings as criteria are well-recognized, yet with nested designs, it is difficult to determine the degree that validity estimates are obscured due to rater variance. Using a simulation methodology, we examined attenuation under different nested validation design characteristics. Results clearly illustrate the attenuating effect that dependencies in nested performance ratings can have on observed criterion-related validity coefficients, and also indicate that attenuation decreases as the number of ratees per rater increases. Furthermore, we also examined observed intraclass correlation coefficients (ICC(1)) as potential alternative corrections to observed validity estimates. If assumptions are met regarding between-rater variability, our results suggest that corrections using a local ICC(1) estimate may be a viable alternative for disattenuating validity coefficients in nested rating designs. Our findings have practical implications for investigators using nested validation designs, for gauging the potential severity of attenuation, designing data collection efforts to minimize attenuation, and for potential post hoc corrections.