Sampling Variability and Axioms of Classical Test Theory

Many well-known equations in classical test theory are mathematical identities in populations of individuals but not in random samples from those populations. First, test scores are subject to the same sampling error that is familiar in statistical estimation and hypothesis testing. Second, the assumptions made in derivation of formulas in test theory are not necessarily satisfied in small samples. The present study derived modified equations relating testscores and components of scores that are identities in samples of any size and that reduce to the more familiar equations when various correlations are zero. Simulations determined the accuracy of both the familiar and the modified equations when applied to samples of various sizes from populations with known reliability coefficients. The programs also determined the variability of the sample values for different parameters in the equations and for different sample sizes, as well as the means and variances of discrepancies between population and sample values.