Using State Assessments to Assign Booklets to NAEP Students to Minimize Measurement Error: An Empirical Study in Four States

Because NAEP estimates of state-level achievement play an important role in the evaluation of strategies for improving the nation's educational system, it is important that these estimates have as small a standard error as possible. NAEP estimates for state-level achievement are based on measurement of the performance of a random sample of students, and the 40-minute measurement of each student's performance includes inherent random error. The standard error of state-level estimates can be reduced either by increasing the sample size, which is expensive, or by reducing the error in each student's measurement. The error of measurement can be reduced by increasing testing time, but that would also entail additional cost, as well as additional burden on the students selected to participate in NAEP. NAEP test booklets each include two blocks of items, and these item blocks vary in difficulty. If booklets with at least one easy block could be targeted to include the lowest achieving students, then the error of measurement for the segment of the population they represent could be reduced at very little cost. Likewise, if booklets with at least one challenging block could be targeted to include the highest achieving students, their measurement error could be reduced. The standard error of NAEP achievement statistics tends to be larger, other things equal, for low achieving groups, so there is a special need for blocks of items that are better fitted to their achievement levels. NAEP has attempted to develop such item blocks since 2000, with limited success. Nevertheless, NAEP item blocks differ in difficulty, and the present study was undertaken to estimate the relation between block difficulty and average standard error of measurement, as well as the resulting reduction in the standard error if students were to be assigned an optimal item block based on the measurement of their achievement provided by state assessment scores. For this study, the records of participants in the 2003 NAEP reading and mathematics assessments in four states were matched to state assessment records, and the standard errors of lowest and highest quartile students, based on the state assessments, were compared for all of the existing NAEP item blocks. Five research questions guided this study: (1) How different are the difficulties of blocks on existing NAEP reading and mathematics assessments? (2) Can state assessments identify potentially low achievers on NAEP? (3) Are standard errors