Improving School Accountability Measures. NBER Working Paper Series

A growing number of states are using annual school-level test scores as part of their school accountability systems. This paper highlights an under-appreciated weakness of that approach, the imprecision of school-level test score means, and proposes a method for discerning signal from noise in annual school report cards. Using methods developed in M. McClellan and D. Staiger (1999) for the analysis of hospital performance measures, the researchers decomposed the variation across schools and over time in test score measures into three components. For an elementary school of average size in North Carolina, it is estimated that 28% of the variance in fifth-grade reading scores results from sampling variation and about 10% results from other nonpersistent sources. A more troubling finding is the estimate that less than half of the variance in the mean gain in reading performance between fourth and fifth grade is due to persistent differences between schools. These estimates of the variance components are used in an empirical Bayes framework to generate "filtered" predictions of school performance, which have much greater predictive value than the mean for a single year. The paper also identifies evidence of within-school heterogeneity in classroom level gains, a finding that suggests the importance of teacher effects. (Contains 11 tables, 5 figures, and 10 references.) (SLD)