Comparing Results of Systematic Reviews: Parallel Reviews of Research on Repeated Reading

Education and related services are relying increasingly on empirically supported treatments (ESTs), which have been shown to improve student outcomes through rigorous research. Many organizations have developed review systems with guidelines for judging the quality of studies and identifying ESTs. However, little explicit attention has been paid to issues of validity of these review systems. In this study, we used the criteria developed by Horner and colleagues (2005), Gersten and colleagues (2005), and the What Works Clearinghouse (WWC, 2008; Kratochwill et al., 2010) to evaluate the research base on repeated reading. The corpus of literature reviewed was derived from previous narrative literature reviews and meta-analyses that concluded that repeated reading was an effective intervention for improving reading fluency. However, the review systems employed in this study resulted in the conclusion that repeated reading did not have enough high quality research support to be considered an EST. The current reviews relied on strict criteria for the quality of each individual study, whereas the previous reviews and meta-analyses included studies with a wider range of quality. These results demonstrate that systematic reviews that strictly appraise the quality of studies and reject those not meeting standards can be substantially more conservative than other scientific review methods. The finding that these different review methods (narrative, meta-analysis, and systematic) can produce diverging recommendations raises issues of validity for practice recommendations.