A comparison of quasi-experimental methods with data before and after an intervention: an introduction for epidemiologists and a simulation study

Abstract Background As the interest in and use of quasi-experimental methods to evaluate impacts of health policies have dramatically increased in the epidemiological literature, we set out this study to (i) systematically compare several quasi-experimental methods that use data before and after an intervention and contrast their performance within a simulation framework while providing a brief overview of the methods; and (ii) discuss challenges that could arise from using these methods as well as directions for future research in the context of epidemiological applications. Methods We considered single-group designs [pre-post and interrupted time series (ITS)] and multiple-group designs [controlled interrupted time series/difference-in-differences, synthetic control methods (SCMs): traditional SCMs and generalized SCMs]. We assessed performance based on bias and root mean squared error. Results We identified settings in which each method failed to provide unbiased estimates. We found that, among the methods investigated, when data for multiple time points and for multiple control groups are available (multiple-group designs), data-adaptive methods such as the generalized SCM were generally less biased than other methods evaluated in our study. In addition, when all of the included units have been exposed to treatment (single-group designs) and data for a sufficiently long pre-intervention period are available, then the ITS performs very well, provided the underlying model is correctly specified. Conclusions When using a quasi-experimental method using data before and after an intervention, epidemiologists should strive to use, whenever feasible, data-adaptive methods that nest alternative identifying assumptions including relaxing the parallel trend assumption (e.g. generalized SCMs).