Using data sources beyond PubMed has a modest impact on the results of systematic reviews of therapeutic interventions

Abstract Objectives Searching multiple sources when conducting systematic reviews is considered good practice. We aimed to investigate the impact of using sources beyond PubMed in systematic reviews of therapeutic interventions. Study Design and Setting We randomly selected 50 Cochrane reviews that searched the PubMed (or MEDLINE) and EMBASE databases and included a meta-analysis of ≥10 studies. We checked whether each eligible record in each review ( n = 2,700) was retrievable in PubMed and EMBASE. For the first-listed meta-analysis of ≥10 studies in each review, we examined whether excluding studies not found in PubMed affected results. Results A median of one record per review was indexed in EMBASE but not in PubMed; a median of four records per review was not indexed in PubMed or EMBASE. Meta-analyses included a median of 13.5 studies; a median of zero studies per meta-analysis was indexed in EMBASE but not in PubMed; a median of one study per meta-analysis was not indexed in PubMed or EMBASE. Meta-analysis using only PubMed-indexed vs. all available studies led to a different conclusion in a single case (on the basis of conventional criteria for statistical significance). In meta-regression analyses, effects in PubMed- vs. non–PubMed-indexed studies were statistically significantly different in a single data set. Conclusion For systematic reviews of the effects of therapeutic interventions, gains from searching sources beyond PubMed, and from searching EMBASE in particular are modest.