The Implications of Population Changes on Generalization and Study Design

Findings from experimental studies have increasingly been used to inform policy in school settings. Thus far, the populations in many of these studies are typically defined in a cross-sectional context; namely, the populations are defined in the same academic year in which the study took place or the population is defined at a fixed time point. This study assesses the extent to which the composition (observable characteristics) of a population changes over time and the implications of these changes for generalization and study design. We center our discussion around two case studies, both of which were cluster randomized trials in education. For each study, we collected multiple years of population level data and analyzed the types of changes that occurred in each study population over the given time period. Overall, we find that the most consistent changes are associated with the racial and demographic composition of the student populations. We discuss the implications of these changes and how they affect the types of populations for whom the findings of a study will be relevant.